Sep 09 23:48:07.685557 ip-10-0-129-102 systemd[1]: Starting Kubernetes Kubelet... Sep 09 23:48:08.042122 ip-10-0-129-102 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:48:08.042122 ip-10-0-129-102 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 23:48:08.042122 ip-10-0-129-102 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:48:08.042122 ip-10-0-129-102 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 23:48:08.042122 ip-10-0-129-102 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:48:08.042737 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.042132 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 23:48:08.048120 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048105 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:48:08.048120 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048119 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048123 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048126 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048129 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048132 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048135 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048138 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048140 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048143 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048146 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048149 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048152 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048162 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048165 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048168 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048170 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048173 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048176 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048179 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048181 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:48:08.048188 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048184 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048187 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048189 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048194 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048197 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048200 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048203 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048205 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048208 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048211 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048213 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048218 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048222 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048225 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048228 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048230 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048233 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048235 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048238 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:48:08.048703 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048240 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048243 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048246 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048249 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048251 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048253 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048256 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048259 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048261 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048264 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048267 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048269 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048272 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048274 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048277 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048293 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048298 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048302 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048305 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048308 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:48:08.049167 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048311 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048313 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048316 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048320 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048324 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048327 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048330 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048332 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048335 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048338 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048341 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048343 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048346 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048348 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048351 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048353 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048356 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048359 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048362 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:48:08.049658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048365 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048368 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048370 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048373 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048376 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048378 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048381 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048384 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048388 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048391 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048393 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048396 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048398 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048401 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048404 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048406 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048409 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048412 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048414 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.048417 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:48:08.050121 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049099 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049107 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049109 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049112 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049115 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049117 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049120 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049123 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049126 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049128 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049131 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049133 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049136 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049140 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049142 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049145 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049147 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049150 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049152 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049155 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049158 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:48:08.050598 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049161 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049164 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049166 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049169 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049172 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049174 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049178 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049181 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049183 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049186 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049190 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049193 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049196 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049199 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049201 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049204 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049207 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049209 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049212 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049214 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:48:08.051261 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049217 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049220 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049222 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049225 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049227 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049230 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049233 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049235 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049238 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049240 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049243 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049249 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049253 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049257 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049261 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049264 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049266 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049269 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049272 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:48:08.051993 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049275 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049277 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049293 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049296 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049299 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049301 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049304 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049306 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049309 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049312 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049314 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049317 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049319 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049322 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049325 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049327 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049330 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049332 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049335 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:48:08.052474 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049338 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049341 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049343 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049347 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049350 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049355 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049359 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049361 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049364 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049367 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049370 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049372 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049375 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049377 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049380 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049383 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049385 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049388 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049390 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:48:08.052997 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049392 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049750 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049754 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049757 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049760 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049763 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049766 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049769 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049772 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049775 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049777 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049780 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049782 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049785 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049788 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049790 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049793 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049796 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049800 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049802 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:48:08.053658 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049805 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049808 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049810 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049813 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049815 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049818 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049821 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049823 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049826 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049829 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049832 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049834 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049837 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049839 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049842 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049844 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049847 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049851 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049854 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:48:08.054140 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049857 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049860 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049863 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049866 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049869 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049872 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049874 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049877 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049880 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049882 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049885 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049889 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049891 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049894 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049896 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049899 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049901 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049904 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049906 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049909 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:48:08.054633 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049911 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049914 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049916 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049919 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049921 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049924 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049927 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049929 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049932 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049934 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049937 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049939 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049941 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049944 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049946 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049949 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049951 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049953 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049956 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:48:08.055116 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049958 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049961 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049963 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049966 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049970 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049972 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049975 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049978 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049980 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049983 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049985 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049988 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049990 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049993 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049995 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.049998 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050000 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050002 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050005 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:48:08.055587 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050009 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050013 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050015 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050085 2574 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050117 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050123 2574 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050127 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050131 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050134 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050139 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050143 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050146 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050149 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050152 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050155 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050160 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050164 2574 flags.go:64] FLAG: --cgroup-root="" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050167 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050170 2574 flags.go:64] FLAG: --client-ca-file="" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050173 2574 flags.go:64] FLAG: --cloud-provider="external" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050176 2574 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050180 2574 flags.go:64] FLAG: --cluster-domain="" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050182 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 23:48:08.056056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050186 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050189 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050192 2574 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050196 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050199 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050202 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050206 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050209 2574 flags.go:64] FLAG: --contention-profiling="false" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050212 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050215 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050218 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050221 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050224 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050227 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050230 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050233 2574 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050236 2574 flags.go:64] FLAG: --enable-server="true" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050239 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050243 2574 flags.go:64] FLAG: --event-burst="100" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050246 2574 flags.go:64] FLAG: --event-qps="50" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050249 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050252 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050255 2574 flags.go:64] FLAG: --eviction-hard="" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050258 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 23:48:08.056615 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050262 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050265 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050269 2574 flags.go:64] FLAG: --eviction-soft="" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050272 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050275 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050293 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050298 2574 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050302 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050307 2574 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050312 2574 flags.go:64] FLAG: --feature-gates="" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050316 2574 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050319 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050322 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050325 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050328 2574 flags.go:64] FLAG: --healthz-port="10248" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050331 2574 flags.go:64] FLAG: --help="false" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050334 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050337 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050340 2574 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050343 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050346 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050349 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050352 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 23:48:08.057219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050355 2574 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050358 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050360 2574 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050363 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050366 2574 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050372 2574 flags.go:64] FLAG: --kube-reserved="" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050374 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050377 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050380 2574 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050383 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050386 2574 flags.go:64] FLAG: --lock-file="" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050390 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050393 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050396 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050402 2574 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050404 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050407 2574 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050410 2574 flags.go:64] FLAG: --logging-format="text" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050413 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050417 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050419 2574 flags.go:64] FLAG: --manifest-url="" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050422 2574 flags.go:64] FLAG: --manifest-url-header="" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050426 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050429 2574 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050433 2574 flags.go:64] FLAG: --max-pods="110" Sep 09 23:48:08.057815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050436 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050439 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050442 2574 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050445 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050448 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050451 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050454 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050460 2574 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050464 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050467 2574 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050470 2574 flags.go:64] FLAG: --pod-cidr="" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050473 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050478 2574 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050482 2574 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050485 2574 flags.go:64] FLAG: --pods-per-core="0" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050488 2574 flags.go:64] FLAG: --port="10250" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050491 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050494 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e57306a0f130ffef" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050497 2574 flags.go:64] FLAG: --qos-reserved="" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050501 2574 flags.go:64] FLAG: --read-only-port="10255" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050504 2574 flags.go:64] FLAG: --register-node="true" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050507 2574 flags.go:64] FLAG: --register-with-taints="" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050510 2574 flags.go:64] FLAG: --registry-burst="10" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050513 2574 flags.go:64] FLAG: --registry-qps="5" Sep 09 23:48:08.058461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050516 2574 flags.go:64] FLAG: --reserved-cpus="" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050519 2574 flags.go:64] FLAG: --reserved-memory="" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050523 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050525 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050529 2574 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050531 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050534 2574 flags.go:64] FLAG: --runonce="false" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050537 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050540 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050544 2574 flags.go:64] FLAG: --seccomp-default="false" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050547 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050550 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050553 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050556 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050559 2574 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050561 2574 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050564 2574 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050567 2574 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050570 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050572 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050575 2574 flags.go:64] FLAG: --system-cgroups="" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050579 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050584 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050588 2574 flags.go:64] FLAG: --tls-cert-file="" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050591 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 23:48:08.059132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050594 2574 flags.go:64] FLAG: --tls-min-version="" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050597 2574 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050600 2574 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050604 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050607 2574 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050610 2574 flags.go:64] FLAG: --v="2" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050614 2574 flags.go:64] FLAG: --version="false" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050618 2574 flags.go:64] FLAG: --vmodule="" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050627 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.050630 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050769 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050775 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050778 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050781 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050783 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050786 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050790 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050793 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050795 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050798 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050801 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050803 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:48:08.059755 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050806 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050808 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050811 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050813 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050816 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050819 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050824 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050827 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050830 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050832 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050835 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050837 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050840 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050843 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.050883 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052827 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052836 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052841 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052845 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:48:08.060319 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052849 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052854 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052859 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052863 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052869 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052873 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052878 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052978 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.052994 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053000 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053003 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053007 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053022 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053028 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053032 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053036 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053040 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053044 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053048 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:48:08.060827 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053055 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053058 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053061 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053064 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053067 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053071 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053074 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053076 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053079 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053082 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053085 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053088 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053090 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053096 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053099 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053101 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053104 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053107 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053110 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053113 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:48:08.061312 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053116 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053130 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053134 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053137 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053140 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053145 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053148 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053150 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053153 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053156 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053159 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053161 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053165 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053167 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053191 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053209 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053213 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053332 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053336 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053339 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:48:08.061794 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053342 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053345 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053347 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053351 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053354 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053356 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053359 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053361 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.053364 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.054057 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.061240 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.061254 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 23:48:08.062269 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.062267 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.062275 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062351 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062357 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062361 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062364 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062366 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062369 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062373 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062375 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062379 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062381 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062384 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062387 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062389 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062392 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062395 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062397 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062400 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:48:08.062584 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062402 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062405 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062408 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062410 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062413 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062416 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062419 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062423 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062427 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062430 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062433 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062436 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062438 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062441 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062444 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062448 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062451 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062454 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062457 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062459 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:48:08.063044 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062462 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062465 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062468 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062470 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062473 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062476 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062478 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062481 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062484 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062486 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062489 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062491 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062495 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062499 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062502 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062505 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062508 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062511 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062514 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062517 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:48:08.063540 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062520 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062523 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062526 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062528 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062531 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062534 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062536 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062539 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062542 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062544 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062547 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062550 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062552 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062555 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062557 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062560 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062563 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062565 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062568 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:48:08.064059 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062571 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062573 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062576 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062578 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062581 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062584 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062587 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062590 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062592 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062595 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062598 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062601 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062603 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062606 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062609 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062611 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062615 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062618 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062621 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:48:08.064524 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062624 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062628 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062631 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062633 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.062638 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062745 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062750 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062752 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062756 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062758 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062762 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062767 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062769 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062772 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:48:08.064987 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062775 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062778 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062780 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062783 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062786 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062788 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062791 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062793 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062796 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062798 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062801 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062804 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062807 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062809 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062812 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062814 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062817 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062819 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062822 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062824 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:48:08.065397 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062828 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062830 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062833 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062835 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062838 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062841 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062843 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062846 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062848 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062851 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062853 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062856 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062858 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062861 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062863 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062866 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062868 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062871 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062873 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062876 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:48:08.065885 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062878 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062882 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062886 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062889 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062891 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062894 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062896 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062899 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062901 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062904 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062906 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062909 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062912 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062915 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062918 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062920 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062923 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062926 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062928 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:48:08.066372 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062932 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062936 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062939 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062942 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062945 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062948 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062950 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062953 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062956 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062958 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062961 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062963 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062965 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062968 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062971 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062973 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062977 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062979 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062982 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062984 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:48:08.066820 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062987 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062989 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062992 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062994 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.062997 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.063000 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.063002 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.063005 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.063008 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.063011 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:08.063013 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.063024 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.063119 2574 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 23:48:08.067300 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.066425 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 23:48:08.067623 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.067361 2574 server.go:1019] "Starting client certificate rotation" Sep 09 23:48:08.067623 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.067459 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 23:48:08.067623 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.067495 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 23:48:08.071547 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.071532 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 23:48:08.072812 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.072791 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 23:48:08.089365 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.089343 2574 log.go:25] "Validated CRI v1 runtime API" Sep 09 23:48:08.092614 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.092599 2574 log.go:25] "Validated CRI v1 image API" Sep 09 23:48:08.093474 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.093458 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 23:48:08.096860 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.096836 2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b923974c-23d2-4768-a4fa-ce803bf334a0:/dev/nvme0n1p4 c39348ee-19f4-4d07-86cb-c00f783862d6:/dev/nvme0n1p3] Sep 09 23:48:08.096939 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.096858 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 23:48:08.096939 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.096883 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 23:48:08.104130 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.104015 2574 manager.go:216] Machine: {Timestamp:2026-09-09 23:48:08.102662527 +0000 UTC m=+0.309802293 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3075711 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aedd03f0e651c9b20bef1487f9882 SystemUUID:ec2aedd0-3f0e-651c-9b20-bef1487f9882 BootID:b4a496ff-c3a6-4d7a-8610-52a6bbf9ffd0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:56:6b:5f:e7:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:56:6b:5f:e7:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:0e:93:fd:68:a3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 23:48:08.104130 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.104119 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 23:48:08.104265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.104189 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 23:48:08.107542 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.107516 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 23:48:08.107666 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.107545 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-102.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 23:48:08.108392 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.108382 2574 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 23:48:08.108430 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.108393 2574 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 23:48:08.108430 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.108401 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 23:48:08.109225 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.109215 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 23:48:08.110380 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.110370 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 23:48:08.114731 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.114720 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 09 23:48:08.114850 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.114839 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 23:48:08.114884 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.114874 2574 kubelet.go:486] "Attempting to sync node with API server" Sep 09 23:48:08.114914 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.114896 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 23:48:08.114942 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.114918 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 23:48:08.114942 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.114932 2574 kubelet.go:398] "Adding apiserver pod source" Sep 09 23:48:08.114999 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.114943 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 23:48:08.115420 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.115405 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 23:48:08.115491 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.115432 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 23:48:08.117961 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.117945 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 23:48:08.120293 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.120256 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hz8mh" Sep 09 23:48:08.120909 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.120893 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 23:48:08.120971 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.120924 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 23:48:08.122273 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122262 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 23:48:08.122332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122291 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 23:48:08.122332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122300 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 23:48:08.122332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122306 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 23:48:08.122332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122314 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 23:48:08.122332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122319 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 23:48:08.122332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122331 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 23:48:08.122492 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122337 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 23:48:08.122492 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122343 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 23:48:08.122492 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122349 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 23:48:08.122492 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122357 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 23:48:08.122492 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.122362 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 23:48:08.123231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.123220 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 23:48:08.123231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.123230 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 23:48:08.123718 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.123698 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 23:48:08.123800 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.123759 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 23:48:08.126627 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.126610 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-102.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 23:48:08.126906 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.126893 2574 server.go:1268] "Started kubelet" Sep 09 23:48:08.126964 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.126934 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hz8mh" Sep 09 23:48:08.127066 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.127009 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 23:48:08.127099 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.127092 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 23:48:08.127423 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.127362 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 23:48:08.127738 ip-10-0-129-102 systemd[1]: Started Kubernetes Kubelet. Sep 09 23:48:08.129616 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.129585 2574 server.go:310] "Adding debug handlers to kubelet server" Sep 09 23:48:08.130449 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.130432 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 23:48:08.135002 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.134984 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 23:48:08.139550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.139531 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 23:48:08.139623 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.139554 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 23:48:08.139814 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.139797 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:08.140450 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140431 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 23:48:08.140540 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140461 2574 factory.go:55] Registering systemd factory Sep 09 23:48:08.140540 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140480 2574 factory.go:223] Registration of the systemd container factory successfully Sep 09 23:48:08.140540 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140487 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 23:48:08.140540 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140505 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 23:48:08.140701 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140594 2574 reconstruct.go:97] "Volume reconstruction finished" Sep 09 23:48:08.140701 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140605 2574 reconciler.go:29] "Reconciler: start to sync state" Sep 09 23:48:08.140701 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140687 2574 factory.go:153] Registering CRI-O factory Sep 09 23:48:08.140701 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140700 2574 factory.go:223] Registration of the crio container factory successfully Sep 09 23:48:08.140891 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140747 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 23:48:08.140891 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140772 2574 factory.go:103] Registering Raw factory Sep 09 23:48:08.140891 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.140784 2574 manager.go:1196] Started watching for new ooms in manager Sep 09 23:48:08.141716 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.141696 2574 manager.go:349] Starting recovery of all containers Sep 09 23:48:08.144200 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.144179 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:48:08.148173 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.147311 2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-102.ec2.internal\" not found" node="ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.158198 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.158182 2574 manager.go:354] Recovery completed Sep 09 23:48:08.163045 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.162947 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:48:08.166829 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.166809 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:48:08.166909 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.166839 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:48:08.166909 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.166853 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:48:08.167439 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.167423 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 23:48:08.167439 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.167438 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 23:48:08.167560 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.167440 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:48:08.167560 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.167461 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 09 23:48:08.168874 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.168861 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:48:08.168938 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.168884 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:48:08.168938 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.168898 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:48:08.169721 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.169708 2574 policy_none.go:49] "None policy: Start" Sep 09 23:48:08.169772 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.169727 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 23:48:08.169772 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.169741 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 23:48:08.171776 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.171763 2574 policy_none.go:47] "Start" Sep 09 23:48:08.198928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.198913 2574 manager.go:337] "Starting Device Plugin manager" Sep 09 23:48:08.216367 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.198946 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 23:48:08.216367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.198958 2574 server.go:87] "Starting device plugin registration server" Sep 09 23:48:08.216367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.199267 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 23:48:08.216367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.199297 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 23:48:08.216367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.199907 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 23:48:08.216367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.199991 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 23:48:08.216367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.200000 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 23:48:08.216367 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.200397 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 23:48:08.216367 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.200445 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:08.231702 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.231674 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 23:48:08.232864 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.232850 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 23:48:08.232941 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.232888 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 23:48:08.232941 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.232907 2574 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 23:48:08.233032 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.232982 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 23:48:08.235396 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.235381 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:48:08.300386 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.300336 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:48:08.301129 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.301113 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:48:08.301205 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.301143 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:48:08.301205 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.301157 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:48:08.301205 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.301191 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.310353 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.310336 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.310409 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.310356 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-102.ec2.internal\": node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:08.322726 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.322704 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:08.333795 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.333775 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal"] Sep 09 23:48:08.333892 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.333832 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:48:08.334589 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.334575 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:48:08.334646 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.334599 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:48:08.334646 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.334609 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:48:08.336736 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.336724 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:48:08.336875 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.336860 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.336918 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.336888 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:48:08.338513 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.338499 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:48:08.338589 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.338516 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:48:08.338589 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.338527 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:48:08.338589 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.338539 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:48:08.338589 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.338541 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:48:08.338589 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.338552 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:48:08.340603 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.340588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.340645 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.340621 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:48:08.341297 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.341262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fd519c173bf3ad3aec8b544d749f3e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal\" (UID: \"7fd519c173bf3ad3aec8b544d749f3e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.341356 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.341308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd519c173bf3ad3aec8b544d749f3e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal\" (UID: \"7fd519c173bf3ad3aec8b544d749f3e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.341356 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.341273 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:48:08.341356 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.341344 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:48:08.341356 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.341353 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:48:08.341521 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.341349 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef2e7d00f6cdedb8b1c2902e991acc2b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-102.ec2.internal\" (UID: \"ef2e7d00f6cdedb8b1c2902e991acc2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.360915 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.360896 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-102.ec2.internal\" not found" node="ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.365330 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.365316 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-102.ec2.internal\" not found" node="ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.423237 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.423215 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:08.441890 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.441866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fd519c173bf3ad3aec8b544d749f3e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal\" (UID: \"7fd519c173bf3ad3aec8b544d749f3e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.441991 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.441896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd519c173bf3ad3aec8b544d749f3e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal\" (UID: \"7fd519c173bf3ad3aec8b544d749f3e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.441991 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.441914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef2e7d00f6cdedb8b1c2902e991acc2b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-102.ec2.internal\" (UID: \"ef2e7d00f6cdedb8b1c2902e991acc2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.441991 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.441917 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fd519c173bf3ad3aec8b544d749f3e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal\" (UID: \"7fd519c173bf3ad3aec8b544d749f3e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.441991 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.441953 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef2e7d00f6cdedb8b1c2902e991acc2b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-102.ec2.internal\" (UID: \"ef2e7d00f6cdedb8b1c2902e991acc2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.441991 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.441953 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd519c173bf3ad3aec8b544d749f3e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal\" (UID: \"7fd519c173bf3ad3aec8b544d749f3e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.524209 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.524173 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:08.624951 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.624889 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:08.663421 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.663387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.668064 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:08.668041 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal" Sep 09 23:48:08.725617 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.725591 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:08.826068 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.826048 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:08.926602 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:08.926546 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:09.027015 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:09.026993 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:09.067503 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.067483 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 23:48:09.067880 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.067636 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 23:48:09.068021 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.067636 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 23:48:09.127168 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:09.127143 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:09.136800 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.136771 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 23:43:08 +0000 UTC" deadline="2028-03-28 00:15:03.300955292 +0000 UTC" Sep 09 23:48:09.136800 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.136798 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13560h26m54.164161364s" Sep 09 23:48:09.201206 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.201149 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:48:09.227356 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:09.227330 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:09.283840 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:09.283810 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2e7d00f6cdedb8b1c2902e991acc2b.slice/crio-469eead23bb1cff6b7af220d9a012b41d8599ce643838299ba18680e048c7533 WatchSource:0}: Error finding container 469eead23bb1cff6b7af220d9a012b41d8599ce643838299ba18680e048c7533: Status 404 returned error can't find the container with id 469eead23bb1cff6b7af220d9a012b41d8599ce643838299ba18680e048c7533 Sep 09 23:48:09.284169 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:09.284146 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd519c173bf3ad3aec8b544d749f3e4.slice/crio-2f39f063b67a7538ceb3479503eb8f78f5cca90a911a1481e09ede943ddaecef WatchSource:0}: Error finding container 2f39f063b67a7538ceb3479503eb8f78f5cca90a911a1481e09ede943ddaecef: Status 404 returned error can't find the container with id 2f39f063b67a7538ceb3479503eb8f78f5cca90a911a1481e09ede943ddaecef Sep 09 23:48:09.287552 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.287535 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:48:09.328927 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:09.328899 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:09.429811 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:09.429785 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:09.536386 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:09.536329 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-102.ec2.internal\" not found" Sep 09 23:48:09.624069 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.624046 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:48:09.640125 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.640104 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" Sep 09 23:48:09.650881 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.650862 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 23:48:09.651778 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.651765 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal" Sep 09 23:48:09.660286 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:09.660272 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 23:48:10.116460 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.116429 2574 apiserver.go:52] "Watching apiserver" Sep 09 23:48:10.124315 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.124286 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 23:48:10.125541 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.125513 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-gmqgc","openshift-network-diagnostics/network-check-target-6mppg","openshift-ovn-kubernetes/ovnkube-node-dwxwt","kube-system/konnectivity-agent-tfn7k","kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm","openshift-cluster-node-tuning-operator/tuned-zbsr7","openshift-image-registry/node-ca-5ksjl","openshift-multus/multus-additional-cni-plugins-cczv2","openshift-network-operator/iptables-alerter-zv62l","kube-system/global-pull-secret-syncer-bjngb","openshift-dns/node-resolver-dmks5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal","openshift-multus/multus-xvjjv"] Sep 09 23:48:10.128179 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.128159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.130548 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.130517 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 23:48:10.131020 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.130665 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:48:10.131020 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.130520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.131020 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.130783 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8m9v2\"" Sep 09 23:48:10.133051 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.132936 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 23:48:10.133266 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.133245 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 23:48:10.134307 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.133475 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 23:48:10.134307 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.133586 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-75hrj\"" Sep 09 23:48:10.134307 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.134275 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.134520 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.134433 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:48:10.136988 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.136526 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 23:48:10.136988 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.136698 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:10.136988 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.136941 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75mhr\"" Sep 09 23:48:10.137183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.137160 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 23:48:10.139171 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.138430 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 23:48:10.139171 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.138433 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 23:48:10.139171 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.138583 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 23:48:10.139171 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.138803 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 23:48:10.139421 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.139227 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 23:48:10.139421 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.139230 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 23:48:10.139421 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.139385 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f9vgb\"" Sep 09 23:48:10.140649 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.140626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:10.140751 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.140696 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 23:48:10.140751 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.140711 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:10.142726 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.142706 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.144749 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.144733 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 23:48:10.145359 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.145328 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 23:48:10.145359 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.145355 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 23:48:10.145533 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.145513 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ggwdm\"" Sep 09 23:48:10.145787 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.145771 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.149661 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.149288 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-modprobe-d\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-sysctl-conf\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-run\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d011011-2cd8-425e-890a-8b4524a1de10-tmp\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-log-socket\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-systemd\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-run-netns\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-run-openvswitch\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-socket-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-sys\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lv7\" (UniqueName: \"kubernetes.io/projected/44d663d3-f7dc-4270-94cc-696d3b55ea0c-kube-api-access-98lv7\") pod \"node-ca-5ksjl\" (UID: \"44d663d3-f7dc-4270-94cc-696d3b55ea0c\") " pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-run-ovn\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-device-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.152993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-sys-fs\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.153305 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.153077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4z5\" (UniqueName: \"kubernetes.io/projected/bd6bc45b-9139-4e6e-ada7-677729814d94-kube-api-access-kn4z5\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.154177 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.153120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-host\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.154177 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.153399 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 23:48:10.154177 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.153614 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2zgxn\"" Sep 09 23:48:10.154345 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-kubernetes\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.154420 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-sysctl-d\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.154475 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-cni-bin\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.154568 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-cni-netd\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.154625 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-ovn-node-metrics-cert\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.154625 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154618 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-ovnkube-script-lib\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.154734 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154660 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9847e1ea-9099-4ae4-a465-50c6a5d86acb-agent-certs\") pod \"konnectivity-agent-tfn7k\" (UID: \"9847e1ea-9099-4ae4-a465-50c6a5d86acb\") " pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:10.154734 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-sysconfig\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.154844 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44d663d3-f7dc-4270-94cc-696d3b55ea0c-host\") pod \"node-ca-5ksjl\" (UID: \"44d663d3-f7dc-4270-94cc-696d3b55ea0c\") " pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.154844 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-node-log\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.154844 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-ovnkube-config\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.154977 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9847e1ea-9099-4ae4-a465-50c6a5d86acb-konnectivity-ca\") pod \"konnectivity-agent-tfn7k\" (UID: \"9847e1ea-9099-4ae4-a465-50c6a5d86acb\") " pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:10.154977 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:10.154977 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154912 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:10.154977 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.154977 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-slash\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155173 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.154971 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:10.155173 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.154989 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-var-lib-openvswitch\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155173 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-etc-openvswitch\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155173 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d011011-2cd8-425e-890a-8b4524a1de10-etc-tuned\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.155173 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrdh\" (UniqueName: \"kubernetes.io/projected/8d011011-2cd8-425e-890a-8b4524a1de10-kube-api-access-wmrdh\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.155173 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44d663d3-f7dc-4270-94cc-696d3b55ea0c-serviceca\") pod \"node-ca-5ksjl\" (UID: \"44d663d3-f7dc-4270-94cc-696d3b55ea0c\") " pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.155173 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155166 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-kubelet\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-systemd-units\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155193 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2dfnb\"" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-run-systemd\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-env-overrides\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155320 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6pz\" (UniqueName: \"kubernetes.io/projected/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-kube-api-access-ht6pz\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pgm4\" (UniqueName: \"kubernetes.io/projected/eb88e26c-b785-40be-b781-088af7d86fe2-kube-api-access-5pgm4\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-registration-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.155471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-lib-modules\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.156039 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.155529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-var-lib-kubelet\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.157724 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.157005 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 23:48:10.157724 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.157224 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 23:48:10.157724 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.157261 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 23:48:10.157724 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.157324 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.157724 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.157515 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 23:48:10.160512 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.160023 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:48:10.160512 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.160038 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 23:48:10.160512 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.160144 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 23:48:10.160512 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.160303 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bmsx4\"" Sep 09 23:48:10.160512 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.160369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.160512 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.160448 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:10.160881 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.160558 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 23:48:10.163193 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.162950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.164815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.164794 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k4qng\"" Sep 09 23:48:10.165341 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.165164 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 23:48:10.165720 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.165556 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 23:48:10.179258 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.179231 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdrrc" Sep 09 23:48:10.188312 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.188286 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdrrc" Sep 09 23:48:10.241633 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.241605 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 23:48:10.242866 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.242811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" event={"ID":"7fd519c173bf3ad3aec8b544d749f3e4","Type":"ContainerStarted","Data":"2f39f063b67a7538ceb3479503eb8f78f5cca90a911a1481e09ede943ddaecef"} Sep 09 23:48:10.245609 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.245583 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal" event={"ID":"ef2e7d00f6cdedb8b1c2902e991acc2b","Type":"ContainerStarted","Data":"469eead23bb1cff6b7af220d9a012b41d8599ce643838299ba18680e048c7533"} Sep 09 23:48:10.256485 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-run-ovn\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.256601 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-device-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.256601 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-var-lib-cni-multus\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.256716 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256590 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-run-ovn\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.256716 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256601 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sntms\" (UniqueName: \"kubernetes.io/projected/e884e0c1-c063-4c91-a1d2-012bc3980235-kube-api-access-sntms\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.256716 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-host\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.256868 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-kubernetes\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.256868 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256773 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-device-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.256868 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-sysctl-d\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.256868 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256803 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-host\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.256868 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-cni-bin\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.256868 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-cni-netd\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256877 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-kubernetes\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-ovnkube-script-lib\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-sysctl-d\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-cni-bin\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55083d6e-c367-40eb-96ad-b70390fc1039-kubelet-config\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-cni-netd\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.256964 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55083d6e-c367-40eb-96ad-b70390fc1039-dbus\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.257064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4z5\" (UniqueName: \"kubernetes.io/projected/bd6bc45b-9139-4e6e-ada7-677729814d94-kube-api-access-kn4z5\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.257094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af-hosts-file\") pod \"node-resolver-dmks5\" (UID: \"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af\") " pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.257231 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.257129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-sysconfig\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.258903 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.258877 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-sysconfig\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.259025 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.258919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-node-log\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.259395 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.259367 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-ovnkube-script-lib\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.259937 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.259915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-ovnkube-config\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.260025 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.259953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:10.260080 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.260132 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260097 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e884e0c1-c063-4c91-a1d2-012bc3980235-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.260192 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-slash\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.260192 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-var-lib-openvswitch\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.260459 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260281 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-slash\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.260459 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d011011-2cd8-425e-890a-8b4524a1de10-etc-tuned\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.260459 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrdh\" (UniqueName: \"kubernetes.io/projected/8d011011-2cd8-425e-890a-8b4524a1de10-kube-api-access-wmrdh\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.260459 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44d663d3-f7dc-4270-94cc-696d3b55ea0c-serviceca\") pod \"node-ca-5ksjl\" (UID: \"44d663d3-f7dc-4270-94cc-696d3b55ea0c\") " pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.260459 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-env-overrides\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.260712 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-system-cni-dir\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.260712 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8b59892-f5c3-4d8f-9950-652dd1c762cb-cni-binary-copy\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.260712 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6pz\" (UniqueName: \"kubernetes.io/projected/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-kube-api-access-ht6pz\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.260854 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-run-k8s-cni-cncf-io\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.260904 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-hostroot\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.260962 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-var-lib-kubelet\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.260962 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-modprobe-d\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.261060 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.260979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-sysctl-conf\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.261060 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-run\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.261060 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d011011-2cd8-425e-890a-8b4524a1de10-tmp\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.261202 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-log-socket\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.261202 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261087 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-var-lib-kubelet\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.261202 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-systemd\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.261202 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-run-netns\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.261202 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af-tmp-dir\") pod \"node-resolver-dmks5\" (UID: \"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af\") " pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.261518 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e884e0c1-c063-4c91-a1d2-012bc3980235-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.261656 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261576 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.261656 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-sys\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.261656 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261652 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lv7\" (UniqueName: \"kubernetes.io/projected/44d663d3-f7dc-4270-94cc-696d3b55ea0c-kube-api-access-98lv7\") pod \"node-ca-5ksjl\" (UID: \"44d663d3-f7dc-4270-94cc-696d3b55ea0c\") " pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.261813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261679 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-sys-fs\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.261813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e884e0c1-c063-4c91-a1d2-012bc3980235-cnibin\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.261813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-var-lib-cni-bin\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.261813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e884e0c1-c063-4c91-a1d2-012bc3980235-system-cni-dir\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.262022 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jb7\" (UniqueName: \"kubernetes.io/projected/a2b8b785-9da1-4d9e-b7f8-11364a210acc-kube-api-access-c6jb7\") pod \"iptables-alerter-zv62l\" (UID: \"a2b8b785-9da1-4d9e-b7f8-11364a210acc\") " pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.262022 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261880 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzjq\" (UniqueName: \"kubernetes.io/projected/c8b59892-f5c3-4d8f-9950-652dd1c762cb-kube-api-access-xxzjq\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.262022 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-ovn-node-metrics-cert\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.262022 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.261962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9847e1ea-9099-4ae4-a465-50c6a5d86acb-agent-certs\") pod \"konnectivity-agent-tfn7k\" (UID: \"9847e1ea-9099-4ae4-a465-50c6a5d86acb\") " pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:10.262022 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:10.262251 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-run-netns\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.262251 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-daemon-config\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.262251 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44d663d3-f7dc-4270-94cc-696d3b55ea0c-host\") pod \"node-ca-5ksjl\" (UID: \"44d663d3-f7dc-4270-94cc-696d3b55ea0c\") " pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.262251 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9847e1ea-9099-4ae4-a465-50c6a5d86acb-konnectivity-ca\") pod \"konnectivity-agent-tfn7k\" (UID: \"9847e1ea-9099-4ae4-a465-50c6a5d86acb\") " pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:10.262251 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e884e0c1-c063-4c91-a1d2-012bc3980235-cni-binary-copy\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.262251 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-cnibin\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.262557 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-conf-dir\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.262557 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-etc-openvswitch\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.262557 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-kubelet\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.262557 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-systemd-units\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.262557 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.262557 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-registration-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.262971 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2b8b785-9da1-4d9e-b7f8-11364a210acc-host-slash\") pod \"iptables-alerter-zv62l\" (UID: \"a2b8b785-9da1-4d9e-b7f8-11364a210acc\") " pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.263030 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.262989 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-run-systemd\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.263091 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.263143 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pgm4\" (UniqueName: \"kubernetes.io/projected/eb88e26c-b785-40be-b781-088af7d86fe2-kube-api-access-5pgm4\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:10.263197 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263139 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e884e0c1-c063-4c91-a1d2-012bc3980235-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.263197 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-socket-dir-parent\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.263295 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263204 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-etc-kubernetes\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.263295 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-lib-modules\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.263295 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263277 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jlc\" (UniqueName: \"kubernetes.io/projected/59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af-kube-api-access-j8jlc\") pod \"node-resolver-dmks5\" (UID: \"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af\") " pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.263465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2b8b785-9da1-4d9e-b7f8-11364a210acc-iptables-alerter-script\") pod \"iptables-alerter-zv62l\" (UID: \"a2b8b785-9da1-4d9e-b7f8-11364a210acc\") " pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.263465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-cni-dir\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.263465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-run-openvswitch\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.263604 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.263604 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.263503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-socket-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.264037 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.264018 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-run-multus-certs\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.264159 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.264142 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-registration-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.264476 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.264455 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-var-lib-openvswitch\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.264574 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.264560 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:10.264655 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.264645 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs podName:eb88e26c-b785-40be-b781-088af7d86fe2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:10.764622927 +0000 UTC m=+2.971762705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs") pod "network-metrics-daemon-gmqgc" (UID: "eb88e26c-b785-40be-b781-088af7d86fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:10.264966 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.264882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-node-log\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.265442 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.265421 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-ovnkube-config\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.265520 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.265497 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.265941 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.265901 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 23:48:10.266472 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.266440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-systemd\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.266576 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.266495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44d663d3-f7dc-4270-94cc-696d3b55ea0c-serviceca\") pod \"node-ca-5ksjl\" (UID: \"44d663d3-f7dc-4270-94cc-696d3b55ea0c\") " pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.266576 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.266551 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-run-netns\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.266683 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.266599 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-sys\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.266880 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.266861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-sys-fs\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.266944 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.266882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-env-overrides\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.267284 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.267263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-run-systemd\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.267381 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.267321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44d663d3-f7dc-4270-94cc-696d3b55ea0c-host\") pod \"node-ca-5ksjl\" (UID: \"44d663d3-f7dc-4270-94cc-696d3b55ea0c\") " pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.267952 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9847e1ea-9099-4ae4-a465-50c6a5d86acb-konnectivity-ca\") pod \"konnectivity-agent-tfn7k\" (UID: \"9847e1ea-9099-4ae4-a465-50c6a5d86acb\") " pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-etc-openvswitch\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-kubelet\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-systemd-units\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268190 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268264 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268376 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-sysctl-conf\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-log-socket\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268447 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-var-lib-kubelet\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268511 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-run\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268644 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-lib-modules\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.268813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.268727 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-run-openvswitch\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.269485 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.269228 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-socket-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.269485 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.269299 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd6bc45b-9139-4e6e-ada7-677729814d94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.269485 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.269346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d011011-2cd8-425e-890a-8b4524a1de10-etc-modprobe-d\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.270315 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.270292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d011011-2cd8-425e-890a-8b4524a1de10-etc-tuned\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.270927 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.270906 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-ovn-node-metrics-cert\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.272483 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.272441 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d011011-2cd8-425e-890a-8b4524a1de10-tmp\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.272671 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.272648 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4z5\" (UniqueName: \"kubernetes.io/projected/bd6bc45b-9139-4e6e-ada7-677729814d94-kube-api-access-kn4z5\") pod \"aws-ebs-csi-driver-node-9pvcm\" (UID: \"bd6bc45b-9139-4e6e-ada7-677729814d94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.272757 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.272649 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9847e1ea-9099-4ae4-a465-50c6a5d86acb-agent-certs\") pod \"konnectivity-agent-tfn7k\" (UID: \"9847e1ea-9099-4ae4-a465-50c6a5d86acb\") " pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:10.279171 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.279139 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrdh\" (UniqueName: \"kubernetes.io/projected/8d011011-2cd8-425e-890a-8b4524a1de10-kube-api-access-wmrdh\") pod \"tuned-zbsr7\" (UID: \"8d011011-2cd8-425e-890a-8b4524a1de10\") " pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.279456 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.279430 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pgm4\" (UniqueName: \"kubernetes.io/projected/eb88e26c-b785-40be-b781-088af7d86fe2-kube-api-access-5pgm4\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:10.279924 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.279892 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lv7\" (UniqueName: \"kubernetes.io/projected/44d663d3-f7dc-4270-94cc-696d3b55ea0c-kube-api-access-98lv7\") pod \"node-ca-5ksjl\" (UID: \"44d663d3-f7dc-4270-94cc-696d3b55ea0c\") " pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.280112 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.280092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6pz\" (UniqueName: \"kubernetes.io/projected/fdda7649-1315-4edf-a2ca-31ae6d15ddbf-kube-api-access-ht6pz\") pod \"ovnkube-node-dwxwt\" (UID: \"fdda7649-1315-4edf-a2ca-31ae6d15ddbf\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.343820 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.343790 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:48:10.364963 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.364936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-run-multus-certs\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365097 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.364970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-var-lib-cni-multus\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365097 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.364992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sntms\" (UniqueName: \"kubernetes.io/projected/e884e0c1-c063-4c91-a1d2-012bc3980235-kube-api-access-sntms\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.365097 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55083d6e-c367-40eb-96ad-b70390fc1039-kubelet-config\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.365097 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365044 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-run-multus-certs\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365097 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365058 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-var-lib-cni-multus\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365097 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55083d6e-c367-40eb-96ad-b70390fc1039-dbus\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55083d6e-c367-40eb-96ad-b70390fc1039-dbus\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55083d6e-c367-40eb-96ad-b70390fc1039-kubelet-config\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af-hosts-file\") pod \"node-resolver-dmks5\" (UID: \"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af\") " pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e884e0c1-c063-4c91-a1d2-012bc3980235-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-system-cni-dir\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8b59892-f5c3-4d8f-9950-652dd1c762cb-cni-binary-copy\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365248 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af-hosts-file\") pod \"node-resolver-dmks5\" (UID: \"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af\") " pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-run-k8s-cni-cncf-io\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-hostroot\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365335 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-run-k8s-cni-cncf-io\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365344 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-system-cni-dir\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-var-lib-kubelet\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-hostroot\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af-tmp-dir\") pod \"node-resolver-dmks5\" (UID: \"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af\") " pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e884e0c1-c063-4c91-a1d2-012bc3980235-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.365478 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e884e0c1-c063-4c91-a1d2-012bc3980235-cnibin\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365403 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-var-lib-kubelet\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-var-lib-cni-bin\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365537 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-var-lib-cni-bin\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e884e0c1-c063-4c91-a1d2-012bc3980235-system-cni-dir\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jb7\" (UniqueName: \"kubernetes.io/projected/a2b8b785-9da1-4d9e-b7f8-11364a210acc-kube-api-access-c6jb7\") pod \"iptables-alerter-zv62l\" (UID: \"a2b8b785-9da1-4d9e-b7f8-11364a210acc\") " pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.365611 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzjq\" (UniqueName: \"kubernetes.io/projected/c8b59892-f5c3-4d8f-9950-652dd1c762cb-kube-api-access-xxzjq\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.365670 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret podName:55083d6e-c367-40eb-96ad-b70390fc1039 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:10.865652286 +0000 UTC m=+3.072792044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret") pod "global-pull-secret-syncer-bjngb" (UID: "55083d6e-c367-40eb-96ad-b70390fc1039") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365694 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e884e0c1-c063-4c91-a1d2-012bc3980235-cnibin\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365725 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e884e0c1-c063-4c91-a1d2-012bc3980235-system-cni-dir\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-run-netns\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-host-run-netns\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-daemon-config\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365814 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e884e0c1-c063-4c91-a1d2-012bc3980235-cni-binary-copy\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-cnibin\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.366301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e884e0c1-c063-4c91-a1d2-012bc3980235-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-conf-dir\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8b59892-f5c3-4d8f-9950-652dd1c762cb-cni-binary-copy\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-conf-dir\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e884e0c1-c063-4c91-a1d2-012bc3980235-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2b8b785-9da1-4d9e-b7f8-11364a210acc-host-slash\") pod \"iptables-alerter-zv62l\" (UID: \"a2b8b785-9da1-4d9e-b7f8-11364a210acc\") " pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-cnibin\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365953 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2b8b785-9da1-4d9e-b7f8-11364a210acc-host-slash\") pod \"iptables-alerter-zv62l\" (UID: \"a2b8b785-9da1-4d9e-b7f8-11364a210acc\") " pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e884e0c1-c063-4c91-a1d2-012bc3980235-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.365991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-socket-dir-parent\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-etc-kubernetes\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jlc\" (UniqueName: \"kubernetes.io/projected/59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af-kube-api-access-j8jlc\") pod \"node-resolver-dmks5\" (UID: \"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af\") " pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2b8b785-9da1-4d9e-b7f8-11364a210acc-iptables-alerter-script\") pod \"iptables-alerter-zv62l\" (UID: \"a2b8b785-9da1-4d9e-b7f8-11364a210acc\") " pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-etc-kubernetes\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-socket-dir-parent\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-cni-dir\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-cni-dir\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366360 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e884e0c1-c063-4c91-a1d2-012bc3980235-cni-binary-copy\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.367006 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366442 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af-tmp-dir\") pod \"node-resolver-dmks5\" (UID: \"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af\") " pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.367665 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8b59892-f5c3-4d8f-9950-652dd1c762cb-multus-daemon-config\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.367665 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366479 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e884e0c1-c063-4c91-a1d2-012bc3980235-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.367665 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.366654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2b8b785-9da1-4d9e-b7f8-11364a210acc-iptables-alerter-script\") pod \"iptables-alerter-zv62l\" (UID: \"a2b8b785-9da1-4d9e-b7f8-11364a210acc\") " pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.371630 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.371608 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:48:10.371756 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.371636 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:48:10.371756 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.371651 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cn2sx for pod openshift-network-diagnostics/network-check-target-6mppg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:10.371756 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.371726 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx podName:dd26812f-6704-4784-98a0-a757cc61c378 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:10.871709288 +0000 UTC m=+3.078849062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cn2sx" (UniqueName: "kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx") pod "network-check-target-6mppg" (UID: "dd26812f-6704-4784-98a0-a757cc61c378") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:10.373677 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.373630 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sntms\" (UniqueName: \"kubernetes.io/projected/e884e0c1-c063-4c91-a1d2-012bc3980235-kube-api-access-sntms\") pod \"multus-additional-cni-plugins-cczv2\" (UID: \"e884e0c1-c063-4c91-a1d2-012bc3980235\") " pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.374197 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.374173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jb7\" (UniqueName: \"kubernetes.io/projected/a2b8b785-9da1-4d9e-b7f8-11364a210acc-kube-api-access-c6jb7\") pod \"iptables-alerter-zv62l\" (UID: \"a2b8b785-9da1-4d9e-b7f8-11364a210acc\") " pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.374452 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.374429 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jlc\" (UniqueName: \"kubernetes.io/projected/59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af-kube-api-access-j8jlc\") pod \"node-resolver-dmks5\" (UID: \"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af\") " pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.374529 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.374511 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzjq\" (UniqueName: \"kubernetes.io/projected/c8b59892-f5c3-4d8f-9950-652dd1c762cb-kube-api-access-xxzjq\") pod \"multus-xvjjv\" (UID: \"c8b59892-f5c3-4d8f-9950-652dd1c762cb\") " pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.443189 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.443149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" Sep 09 23:48:10.456780 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.456754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5ksjl" Sep 09 23:48:10.466638 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.466618 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:10.472326 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.472304 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:10.482193 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.482175 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" Sep 09 23:48:10.487986 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.487967 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cczv2" Sep 09 23:48:10.495758 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.495740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xvjjv" Sep 09 23:48:10.504638 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.504622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zv62l" Sep 09 23:48:10.510375 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.510347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dmks5" Sep 09 23:48:10.769390 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.769283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:10.769561 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.769491 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:10.769630 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.769591 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs podName:eb88e26c-b785-40be-b781-088af7d86fe2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:11.769567831 +0000 UTC m=+3.976707595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs") pod "network-metrics-daemon-gmqgc" (UID: "eb88e26c-b785-40be-b781-088af7d86fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:10.869941 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.869898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:10.870129 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.870088 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:10.870192 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.870164 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret podName:55083d6e-c367-40eb-96ad-b70390fc1039 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:11.870140776 +0000 UTC m=+4.077280539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret") pod "global-pull-secret-syncer-bjngb" (UID: "55083d6e-c367-40eb-96ad-b70390fc1039") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:10.970814 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:10.970778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:10.970985 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.970963 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:48:10.971038 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.970990 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:48:10.971038 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.971000 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cn2sx for pod openshift-network-diagnostics/network-check-target-6mppg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:10.971141 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:10.971062 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx podName:dd26812f-6704-4784-98a0-a757cc61c378 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:11.971047288 +0000 UTC m=+4.178187051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn2sx" (UniqueName: "kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx") pod "network-check-target-6mppg" (UID: "dd26812f-6704-4784-98a0-a757cc61c378") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:11.191660 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:11.191569 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:43:10 +0000 UTC" deadline="2028-03-10 16:59:43.857949661 +0000 UTC" Sep 09 23:48:11.191660 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:11.191604 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13145h11m32.666350125s" Sep 09 23:48:11.371783 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:11.371758 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9847e1ea_9099_4ae4_a465_50c6a5d86acb.slice/crio-a8eea166b9ec6d9bd908a4bed1aa842f415e4337535a4fb4e5a1db0e414f70ff WatchSource:0}: Error finding container a8eea166b9ec6d9bd908a4bed1aa842f415e4337535a4fb4e5a1db0e414f70ff: Status 404 returned error can't find the container with id a8eea166b9ec6d9bd908a4bed1aa842f415e4337535a4fb4e5a1db0e414f70ff Sep 09 23:48:11.373218 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:11.373177 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b59892_f5c3_4d8f_9950_652dd1c762cb.slice/crio-fc0a194508ed2e6f3db7302a415b428cf03b3bf10ff6e694c288c717b5d35f2c WatchSource:0}: Error finding container fc0a194508ed2e6f3db7302a415b428cf03b3bf10ff6e694c288c717b5d35f2c: Status 404 returned error can't find the container with id fc0a194508ed2e6f3db7302a415b428cf03b3bf10ff6e694c288c717b5d35f2c Sep 09 23:48:11.374687 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:11.374664 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b8b785_9da1_4d9e_b7f8_11364a210acc.slice/crio-3395051194fac5e8c0c1aef2f171184f6eb9479c15eef220803f84d7cdfa9450 WatchSource:0}: Error finding container 3395051194fac5e8c0c1aef2f171184f6eb9479c15eef220803f84d7cdfa9450: Status 404 returned error can't find the container with id 3395051194fac5e8c0c1aef2f171184f6eb9479c15eef220803f84d7cdfa9450 Sep 09 23:48:11.377242 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:11.377077 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d663d3_f7dc_4270_94cc_696d3b55ea0c.slice/crio-6ddd063aac52abeae7e8f59c910a77027d421332b1063bb8bf2037696f47cdf5 WatchSource:0}: Error finding container 6ddd063aac52abeae7e8f59c910a77027d421332b1063bb8bf2037696f47cdf5: Status 404 returned error can't find the container with id 6ddd063aac52abeae7e8f59c910a77027d421332b1063bb8bf2037696f47cdf5 Sep 09 23:48:11.378340 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:11.378242 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d011011_2cd8_425e_890a_8b4524a1de10.slice/crio-6bbb5ca1c15bc71ce1d1749f45ccd862fd894a47a490535cea5b1cb8e657831c WatchSource:0}: Error finding container 6bbb5ca1c15bc71ce1d1749f45ccd862fd894a47a490535cea5b1cb8e657831c: Status 404 returned error can't find the container with id 6bbb5ca1c15bc71ce1d1749f45ccd862fd894a47a490535cea5b1cb8e657831c Sep 09 23:48:11.379917 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:11.379528 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6bc45b_9139_4e6e_ada7_677729814d94.slice/crio-1514605bfbf4ac14086f15c157d0bf35887b77a76c2cad9eedd5520491bdf35a WatchSource:0}: Error finding container 1514605bfbf4ac14086f15c157d0bf35887b77a76c2cad9eedd5520491bdf35a: Status 404 returned error can't find the container with id 1514605bfbf4ac14086f15c157d0bf35887b77a76c2cad9eedd5520491bdf35a Sep 09 23:48:11.777439 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:11.777151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:11.777439 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:11.777380 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:11.777647 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:11.777470 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs podName:eb88e26c-b785-40be-b781-088af7d86fe2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:13.777447672 +0000 UTC m=+5.984587437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs") pod "network-metrics-daemon-gmqgc" (UID: "eb88e26c-b785-40be-b781-088af7d86fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:11.878490 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:11.878442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:11.878675 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:11.878607 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:11.878675 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:11.878675 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret podName:55083d6e-c367-40eb-96ad-b70390fc1039 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:13.878656707 +0000 UTC m=+6.085796479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret") pod "global-pull-secret-syncer-bjngb" (UID: "55083d6e-c367-40eb-96ad-b70390fc1039") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:11.980049 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:11.980007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:11.980240 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:11.980216 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:48:11.980313 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:11.980251 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:48:11.980313 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:11.980266 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cn2sx for pod openshift-network-diagnostics/network-check-target-6mppg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:11.980439 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:11.980357 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx podName:dd26812f-6704-4784-98a0-a757cc61c378 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:13.980337173 +0000 UTC m=+6.187476937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn2sx" (UniqueName: "kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx") pod "network-check-target-6mppg" (UID: "dd26812f-6704-4784-98a0-a757cc61c378") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:12.191964 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.191811 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:43:10 +0000 UTC" deadline="2028-03-30 09:53:49.318969374 +0000 UTC" Sep 09 23:48:12.191964 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.191876 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13618h5m37.127097468s" Sep 09 23:48:12.237145 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.236547 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:12.237145 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:12.236693 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:12.237145 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.237093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:12.237481 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:12.237189 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:12.237481 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.237093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:12.237481 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:12.237277 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:12.258962 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.258930 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" event={"ID":"8d011011-2cd8-425e-890a-8b4524a1de10","Type":"ContainerStarted","Data":"6bbb5ca1c15bc71ce1d1749f45ccd862fd894a47a490535cea5b1cb8e657831c"} Sep 09 23:48:12.265773 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.265744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5ksjl" event={"ID":"44d663d3-f7dc-4270-94cc-696d3b55ea0c","Type":"ContainerStarted","Data":"6ddd063aac52abeae7e8f59c910a77027d421332b1063bb8bf2037696f47cdf5"} Sep 09 23:48:12.269798 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.269727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tfn7k" event={"ID":"9847e1ea-9099-4ae4-a465-50c6a5d86acb","Type":"ContainerStarted","Data":"a8eea166b9ec6d9bd908a4bed1aa842f415e4337535a4fb4e5a1db0e414f70ff"} Sep 09 23:48:12.281791 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.281729 2574 generic.go:356] "Generic (PLEG): container finished" podID="7fd519c173bf3ad3aec8b544d749f3e4" containerID="d32c033263a02beba453464e81316b8d3ca107be0627c670333331cad58c0dfd" exitCode=0 Sep 09 23:48:12.282274 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.282247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" event={"ID":"7fd519c173bf3ad3aec8b544d749f3e4","Type":"ContainerDied","Data":"d32c033263a02beba453464e81316b8d3ca107be0627c670333331cad58c0dfd"} Sep 09 23:48:12.286825 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.286798 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dmks5" event={"ID":"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af","Type":"ContainerStarted","Data":"07180e493693ec5e3ae4f0ab1a87c869fc1da361a8db7cd1a710f6589e581127"} Sep 09 23:48:12.289070 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.289044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" event={"ID":"bd6bc45b-9139-4e6e-ada7-677729814d94","Type":"ContainerStarted","Data":"1514605bfbf4ac14086f15c157d0bf35887b77a76c2cad9eedd5520491bdf35a"} Sep 09 23:48:12.306035 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.306004 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zv62l" event={"ID":"a2b8b785-9da1-4d9e-b7f8-11364a210acc","Type":"ContainerStarted","Data":"3395051194fac5e8c0c1aef2f171184f6eb9479c15eef220803f84d7cdfa9450"} Sep 09 23:48:12.309921 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.309836 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xvjjv" event={"ID":"c8b59892-f5c3-4d8f-9950-652dd1c762cb","Type":"ContainerStarted","Data":"fc0a194508ed2e6f3db7302a415b428cf03b3bf10ff6e694c288c717b5d35f2c"} Sep 09 23:48:12.318918 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.318888 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal" event={"ID":"ef2e7d00f6cdedb8b1c2902e991acc2b","Type":"ContainerStarted","Data":"9df75196631833b6a1e78dddfb6bd526cd21624cb7b1bf41d80055c48b5be7dc"} Sep 09 23:48:12.345217 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.345186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerStarted","Data":"07ebc8e9d356cfaaf6f180ea7d3a0b6770f58b619b8d0516b19fc7a1092548c7"} Sep 09 23:48:12.370612 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:12.370551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cczv2" event={"ID":"e884e0c1-c063-4c91-a1d2-012bc3980235","Type":"ContainerStarted","Data":"09f2cc0382d1e480e680985712aef729b4a3fd44595f5b5b527423fe264a85d3"} Sep 09 23:48:13.399093 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:13.398830 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" event={"ID":"7fd519c173bf3ad3aec8b544d749f3e4","Type":"ContainerStarted","Data":"16e8634c1a31c497242c749def18a9831fbf2822d594db36b3b8eecb5181ed67"} Sep 09 23:48:13.416498 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:13.415341 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-102.ec2.internal" podStartSLOduration=3.41531933 podStartE2EDuration="3.41531933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:12.333805739 +0000 UTC m=+4.540945512" watchObservedRunningTime="2026-09-09 23:48:13.41531933 +0000 UTC m=+5.622459102" Sep 09 23:48:13.797368 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:13.796730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:13.797368 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:13.796883 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:13.797368 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:13.796949 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs podName:eb88e26c-b785-40be-b781-088af7d86fe2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:17.796930576 +0000 UTC m=+10.004070340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs") pod "network-metrics-daemon-gmqgc" (UID: "eb88e26c-b785-40be-b781-088af7d86fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:13.897958 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:13.897918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:13.898153 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:13.898083 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:13.898153 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:13.898145 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret podName:55083d6e-c367-40eb-96ad-b70390fc1039 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:17.898127299 +0000 UTC m=+10.105267063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret") pod "global-pull-secret-syncer-bjngb" (UID: "55083d6e-c367-40eb-96ad-b70390fc1039") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:13.998367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:13.998315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:13.998544 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:13.998527 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:48:13.998607 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:13.998552 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:48:13.998607 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:13.998566 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cn2sx for pod openshift-network-diagnostics/network-check-target-6mppg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:13.998791 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:13.998627 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx podName:dd26812f-6704-4784-98a0-a757cc61c378 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:17.998607829 +0000 UTC m=+10.205747596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn2sx" (UniqueName: "kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx") pod "network-check-target-6mppg" (UID: "dd26812f-6704-4784-98a0-a757cc61c378") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:14.235314 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:14.234453 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:14.235314 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:14.234601 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:14.235314 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:14.234990 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:14.235314 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:14.235089 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:14.235314 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:14.235144 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:14.235314 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:14.235219 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:16.234155 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:16.234119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:16.234734 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:16.234258 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:16.234734 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:16.234367 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:16.234734 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:16.234447 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:16.234734 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:16.234592 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:16.234734 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:16.234689 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:17.827095 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:17.827060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:17.827514 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:17.827200 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:17.827514 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:17.827296 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs podName:eb88e26c-b785-40be-b781-088af7d86fe2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:25.827258063 +0000 UTC m=+18.034397832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs") pod "network-metrics-daemon-gmqgc" (UID: "eb88e26c-b785-40be-b781-088af7d86fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:17.927630 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:17.927597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:17.927800 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:17.927753 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:17.927870 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:17.927823 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret podName:55083d6e-c367-40eb-96ad-b70390fc1039 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:25.927809642 +0000 UTC m=+18.134949392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret") pod "global-pull-secret-syncer-bjngb" (UID: "55083d6e-c367-40eb-96ad-b70390fc1039") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:18.028388 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:18.028308 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:18.028545 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:18.028468 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:48:18.028545 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:18.028490 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:48:18.028545 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:18.028501 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cn2sx for pod openshift-network-diagnostics/network-check-target-6mppg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:18.028707 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:18.028564 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx podName:dd26812f-6704-4784-98a0-a757cc61c378 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:26.028545311 +0000 UTC m=+18.235685059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn2sx" (UniqueName: "kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx") pod "network-check-target-6mppg" (UID: "dd26812f-6704-4784-98a0-a757cc61c378") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:18.235736 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:18.234971 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:18.235736 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:18.235085 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:18.235736 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:18.235431 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:18.235736 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:18.235521 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:18.235736 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:18.235568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:18.235736 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:18.235634 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:20.234473 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:20.233998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:20.234473 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:20.234045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:20.234473 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:20.233998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:20.234473 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:20.234110 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:20.234473 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:20.234212 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:20.234473 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:20.234303 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:22.233511 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:22.233483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:22.233960 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:22.233522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:22.233960 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:22.233483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:22.233960 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:22.233588 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:22.233960 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:22.233687 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:22.233960 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:22.233770 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:24.233502 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:24.233464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:24.233945 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:24.233469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:24.233945 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:24.233599 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:24.233945 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:24.233473 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:24.233945 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:24.233673 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:24.233945 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:24.233766 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:25.881541 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:25.881455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:25.882011 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:25.881587 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:25.882011 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:25.881675 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs podName:eb88e26c-b785-40be-b781-088af7d86fe2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:41.881658095 +0000 UTC m=+34.088797846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs") pod "network-metrics-daemon-gmqgc" (UID: "eb88e26c-b785-40be-b781-088af7d86fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:25.982435 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:25.982403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:25.982600 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:25.982566 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:25.982667 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:25.982637 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret podName:55083d6e-c367-40eb-96ad-b70390fc1039 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:41.982616333 +0000 UTC m=+34.189756094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret") pod "global-pull-secret-syncer-bjngb" (UID: "55083d6e-c367-40eb-96ad-b70390fc1039") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:26.083544 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:26.083507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:26.083703 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:26.083669 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:48:26.083703 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:26.083696 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:48:26.083824 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:26.083708 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cn2sx for pod openshift-network-diagnostics/network-check-target-6mppg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:26.083824 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:26.083761 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx podName:dd26812f-6704-4784-98a0-a757cc61c378 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:42.083749183 +0000 UTC m=+34.290888933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn2sx" (UniqueName: "kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx") pod "network-check-target-6mppg" (UID: "dd26812f-6704-4784-98a0-a757cc61c378") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:26.236219 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:26.236153 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:26.236378 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:26.236153 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:26.236378 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:26.236263 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:26.236378 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:26.236159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:26.236528 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:26.236380 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:26.236528 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:26.236458 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:28.237042 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:28.236820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:28.237459 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:28.236874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:28.237459 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:28.237125 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:28.237459 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:28.236886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:28.237459 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:28.237211 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:28.237459 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:28.237330 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:29.434360 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.434323 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerStarted","Data":"db8cf0f11dd5f313515d01b813db8bd0db4dfa68223c637543053a52cb012b96"} Sep 09 23:48:29.435745 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.435720 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cczv2" event={"ID":"e884e0c1-c063-4c91-a1d2-012bc3980235","Type":"ContainerStarted","Data":"af236f568273006000c934fe5fcf8170383f10f4b8ae0db8f1b4266e5767c055"} Sep 09 23:48:29.439597 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.439573 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" event={"ID":"8d011011-2cd8-425e-890a-8b4524a1de10","Type":"ContainerStarted","Data":"dab6c67314a04f9ff139bfd939151acec8a12568c5bde3ab40872c1b5eeeab54"} Sep 09 23:48:29.441737 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.441642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5ksjl" event={"ID":"44d663d3-f7dc-4270-94cc-696d3b55ea0c","Type":"ContainerStarted","Data":"993bc84ecc4225ce88bfe5b6995b8c8b9ef6fc0836b22ff00dee1ac2d10a51c7"} Sep 09 23:48:29.443577 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.443558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tfn7k" event={"ID":"9847e1ea-9099-4ae4-a465-50c6a5d86acb","Type":"ContainerStarted","Data":"02ba10ff348a0e91fa910eacdc70ce81458a8e22ac05a365c2758c470e35d11e"} Sep 09 23:48:29.445269 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.445247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dmks5" event={"ID":"59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af","Type":"ContainerStarted","Data":"00a183e9a1483b4675c6e7dd75337d2d5d7fcae0545c4a2eb782413c9048f019"} Sep 09 23:48:29.447273 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.447225 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" event={"ID":"bd6bc45b-9139-4e6e-ada7-677729814d94","Type":"ContainerStarted","Data":"d9f5b01bd8679940cfad68d7f65daa8925903cdbcbeff86ce87ed9161910f985"} Sep 09 23:48:29.457882 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.457841 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-102.ec2.internal" podStartSLOduration=19.457829618 podStartE2EDuration="19.457829618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:13.416266181 +0000 UTC m=+5.623405956" watchObservedRunningTime="2026-09-09 23:48:29.457829618 +0000 UTC m=+21.664969387" Sep 09 23:48:29.469732 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.469380 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tfn7k" podStartSLOduration=7.428304643 podStartE2EDuration="21.469366572s" totalImagesPullingTime="14.041061929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:29.469214059 +0000 UTC m=+21.676353829" watchObservedRunningTime="2026-09-09 23:48:29.469366572 +0000 UTC m=+21.676506342" Sep 09 23:48:29.485951 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.485747 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zbsr7" podStartSLOduration=3.698624698 podStartE2EDuration="21.485728928s" totalImagesPullingTime="17.78710423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:29.485242949 +0000 UTC m=+21.692382718" watchObservedRunningTime="2026-09-09 23:48:29.485728928 +0000 UTC m=+21.692868698" Sep 09 23:48:29.502526 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:29.502472 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5ksjl" podStartSLOduration=3.824188405 podStartE2EDuration="21.502450253s" totalImagesPullingTime="17.678261848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:29.502255208 +0000 UTC m=+21.709394988" watchObservedRunningTime="2026-09-09 23:48:29.502450253 +0000 UTC m=+21.709590022" Sep 09 23:48:30.233569 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.233536 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:30.233725 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.233588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:30.233725 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.233536 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:30.233725 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:30.233670 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:30.233876 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:30.233762 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:30.233876 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:30.233851 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:30.452251 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.452224 2574 generic.go:356] "Generic (PLEG): container finished" podID="e884e0c1-c063-4c91-a1d2-012bc3980235" containerID="af236f568273006000c934fe5fcf8170383f10f4b8ae0db8f1b4266e5767c055" exitCode=0 Sep 09 23:48:30.452976 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.452313 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cczv2" event={"ID":"e884e0c1-c063-4c91-a1d2-012bc3980235","Type":"ContainerDied","Data":"af236f568273006000c934fe5fcf8170383f10f4b8ae0db8f1b4266e5767c055"} Sep 09 23:48:30.453748 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.453723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zv62l" event={"ID":"a2b8b785-9da1-4d9e-b7f8-11364a210acc","Type":"ContainerStarted","Data":"7c8b562e3843dd66c25f30c012797de78eb9709d49f68dafc1e73227091c3e4c"} Sep 09 23:48:30.455335 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.455311 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xvjjv" event={"ID":"c8b59892-f5c3-4d8f-9950-652dd1c762cb","Type":"ContainerStarted","Data":"1d52d892c54ff0f0e20f792f12c6a2a692d396943cd71e475d9b71faff99103e"} Sep 09 23:48:30.458023 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.458008 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 09 23:48:30.458344 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.458327 2574 generic.go:356] "Generic (PLEG): container finished" podID="fdda7649-1315-4edf-a2ca-31ae6d15ddbf" containerID="112f77af98c9cafbc1e3f03e0d6cbf82f0269c2f9172f6745181e94dd9688054" exitCode=1 Sep 09 23:48:30.458424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.458407 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerStarted","Data":"0045363b903a42efbccec57d8addfea01ed735a40df5199a7297a479c530f348"} Sep 09 23:48:30.458476 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.458464 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerStarted","Data":"4b9905fae414607dcd4c001ae41a35f5949000257e23a58c00a398442488747b"} Sep 09 23:48:30.458517 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.458498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerStarted","Data":"9c5f9d88a2b772e3a0ec3b5f8afa3d3041c2b306ae772ccd59a1f3cd4d975fdf"} Sep 09 23:48:30.458556 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.458535 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerStarted","Data":"a364eedf1620eca3d1fda85279c5f53fad5a5c57a7289a70a088de10b901cffc"} Sep 09 23:48:30.458604 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.458563 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerDied","Data":"112f77af98c9cafbc1e3f03e0d6cbf82f0269c2f9172f6745181e94dd9688054"} Sep 09 23:48:30.471952 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.471913 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dmks5" podStartSLOduration=4.703803064 podStartE2EDuration="22.471902091s" totalImagesPullingTime="17.768099027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:29.523945807 +0000 UTC m=+21.731085575" watchObservedRunningTime="2026-09-09 23:48:30.471902091 +0000 UTC m=+22.679041907" Sep 09 23:48:30.503179 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.503143 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zv62l" podStartSLOduration=4.822848848 podStartE2EDuration="22.503131066s" totalImagesPullingTime="17.680282218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:30.502964742 +0000 UTC m=+22.710104512" watchObservedRunningTime="2026-09-09 23:48:30.503131066 +0000 UTC m=+22.710270835" Sep 09 23:48:30.503309 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.503272 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xvjjv" podStartSLOduration=4.510477777 podStartE2EDuration="22.503266609s" totalImagesPullingTime="17.992788832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:30.49010943 +0000 UTC m=+22.697249210" watchObservedRunningTime="2026-09-09 23:48:30.503266609 +0000 UTC m=+22.710406389" Sep 09 23:48:30.684357 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:30.684302 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 23:48:31.212705 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:31.212495 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T23:48:30.684322031Z","UUID":"20399196-43b6-4b39-b154-4b10a59e89aa","Handler":null,"Name":"","Endpoint":""} Sep 09 23:48:31.214492 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:31.214462 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 23:48:31.214607 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:31.214499 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 23:48:31.463309 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:31.463219 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" event={"ID":"bd6bc45b-9139-4e6e-ada7-677729814d94","Type":"ContainerStarted","Data":"3c5859fdd63a56622ed1d724cf5d51278ccc4df1785602e905247f993846b09d"} Sep 09 23:48:32.236979 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:32.236766 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:32.237142 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:32.236766 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:32.237142 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:32.237016 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:32.237142 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:32.236782 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:32.237323 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:32.237103 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:32.237323 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:32.237227 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:32.468214 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:32.468170 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" event={"ID":"bd6bc45b-9139-4e6e-ada7-677729814d94","Type":"ContainerStarted","Data":"7640271938e8288594c2081b47663fca68b0e6a00222241d92935d8b967c8b59"} Sep 09 23:48:32.471541 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:32.471521 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 09 23:48:32.471932 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:32.471908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerStarted","Data":"1e4f2b6943972989a652e0f687636014aa7133dd7d62493f2d0e705cf83a39cf"} Sep 09 23:48:32.485541 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:32.485493 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pvcm" podStartSLOduration=4.717862597 podStartE2EDuration="24.485476258s" totalImagesPullingTime="19.767613661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:32.485043012 +0000 UTC m=+24.692182782" watchObservedRunningTime="2026-09-09 23:48:32.485476258 +0000 UTC m=+24.692616043" Sep 09 23:48:33.051185 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:33.051150 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:33.519073 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:33.519003 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:33.519608 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:33.519587 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:34.236446 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:34.236371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:34.236601 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:34.236485 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:34.236601 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:34.236499 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:34.236601 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:34.236519 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:34.236745 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:34.236613 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:34.236745 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:34.236700 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:34.477571 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:34.477544 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tfn7k" Sep 09 23:48:35.483118 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:35.482902 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 09 23:48:35.483526 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:35.483478 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerStarted","Data":"92365a11837c3537bf687ce3b67321539edcba5d93603b0a3474c63e519f5b5f"} Sep 09 23:48:35.483947 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:35.483927 2574 scope.go:117] "RemoveContainer" containerID="112f77af98c9cafbc1e3f03e0d6cbf82f0269c2f9172f6745181e94dd9688054" Sep 09 23:48:36.233822 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.233796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:36.233946 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.233796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:36.233946 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:36.233909 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:36.234047 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.233796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:36.234047 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:36.233991 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:36.234112 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:36.234083 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:36.491173 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.491089 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 09 23:48:36.491800 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.491572 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" event={"ID":"fdda7649-1315-4edf-a2ca-31ae6d15ddbf","Type":"ContainerStarted","Data":"9f4eed15bf8d828203d72aed0e7e6ee9a56754b36c1a057d173d1d32bd327038"} Sep 09 23:48:36.491800 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.491657 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 23:48:36.492003 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.491984 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:36.492066 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.492060 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:36.494059 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.494023 2574 generic.go:356] "Generic (PLEG): container finished" podID="e884e0c1-c063-4c91-a1d2-012bc3980235" containerID="0258517d4ce210478faa5834ea3d18aca3480ce59eb778f0b24f004064b817be" exitCode=0 Sep 09 23:48:36.494194 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.494082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cczv2" event={"ID":"e884e0c1-c063-4c91-a1d2-012bc3980235","Type":"ContainerDied","Data":"0258517d4ce210478faa5834ea3d18aca3480ce59eb778f0b24f004064b817be"} Sep 09 23:48:36.510612 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.510592 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:36.510907 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.510887 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:36.523608 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:36.523552 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" podStartSLOduration=10.703745332 podStartE2EDuration="28.523534037s" totalImagesPullingTime="17.819788705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:36.522195505 +0000 UTC m=+28.729335274" watchObservedRunningTime="2026-09-09 23:48:36.523534037 +0000 UTC m=+28.730673812" Sep 09 23:48:37.183779 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:37.183708 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6mppg"] Sep 09 23:48:37.183927 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:37.183863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:37.183989 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:37.183964 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:37.186652 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:37.186623 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjngb"] Sep 09 23:48:37.186783 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:37.186754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:37.186906 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:37.186874 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:37.187452 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:37.187425 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gmqgc"] Sep 09 23:48:37.187554 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:37.187528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:37.187653 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:37.187630 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:37.496757 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:37.496696 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 23:48:38.233859 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:38.233708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:38.234037 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:38.233904 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:38.501494 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:38.501385 2574 generic.go:356] "Generic (PLEG): container finished" podID="e884e0c1-c063-4c91-a1d2-012bc3980235" containerID="6fe219ab4c5756d73a7cd841b936974634bba97a0a9f405f17a4f29046775d54" exitCode=0 Sep 09 23:48:38.501494 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:38.501468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cczv2" event={"ID":"e884e0c1-c063-4c91-a1d2-012bc3980235","Type":"ContainerDied","Data":"6fe219ab4c5756d73a7cd841b936974634bba97a0a9f405f17a4f29046775d54"} Sep 09 23:48:38.501961 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:38.501522 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 23:48:39.233795 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:39.233725 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:39.233923 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:39.233841 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:39.233923 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:39.233883 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:39.233990 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:39.233969 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:39.551680 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:39.551612 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:48:40.236368 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:40.236345 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:40.236517 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:40.236430 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:40.507540 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:40.507470 2574 generic.go:356] "Generic (PLEG): container finished" podID="e884e0c1-c063-4c91-a1d2-012bc3980235" containerID="6a64bd94eb083c649c843156ddde9df4c6b4a4f1cda2d64cd78a3d45d9c54d7f" exitCode=0 Sep 09 23:48:40.507540 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:40.507520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cczv2" event={"ID":"e884e0c1-c063-4c91-a1d2-012bc3980235","Type":"ContainerDied","Data":"6a64bd94eb083c649c843156ddde9df4c6b4a4f1cda2d64cd78a3d45d9c54d7f"} Sep 09 23:48:41.234074 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:41.234005 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:41.234507 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:41.234005 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:41.234507 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:41.234106 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:41.234507 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:41.234195 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:41.903787 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:41.903750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:41.903972 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:41.903920 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:41.904039 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:41.903987 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs podName:eb88e26c-b785-40be-b781-088af7d86fe2 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:13.903972027 +0000 UTC m=+66.111111787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs") pod "network-metrics-daemon-gmqgc" (UID: "eb88e26c-b785-40be-b781-088af7d86fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:48:42.004319 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:42.004269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:42.004475 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:42.004400 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:42.004557 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:42.004540 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret podName:55083d6e-c367-40eb-96ad-b70390fc1039 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:14.004490613 +0000 UTC m=+66.211630375 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret") pod "global-pull-secret-syncer-bjngb" (UID: "55083d6e-c367-40eb-96ad-b70390fc1039") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:48:42.105294 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:42.105263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:42.105429 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:42.105398 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:48:42.105429 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:42.105411 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:48:42.105429 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:42.105421 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cn2sx for pod openshift-network-diagnostics/network-check-target-6mppg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:42.105544 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:42.105464 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx podName:dd26812f-6704-4784-98a0-a757cc61c378 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:14.105450632 +0000 UTC m=+66.312590393 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cn2sx" (UniqueName: "kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx") pod "network-check-target-6mppg" (UID: "dd26812f-6704-4784-98a0-a757cc61c378") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:48:42.233990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:42.233924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:42.234136 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:42.234050 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:43.234133 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:43.234099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:43.234663 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:43.234207 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:43.234663 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:43.234297 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:43.234663 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:43.234411 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:44.233997 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:44.233965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:44.234182 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:44.234095 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:45.233509 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:45.233486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:45.233606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:45.233512 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:45.233606 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:45.233582 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:45.233789 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:45.233755 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:45.521456 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:45.521377 2574 generic.go:356] "Generic (PLEG): container finished" podID="e884e0c1-c063-4c91-a1d2-012bc3980235" containerID="8a79b388102a865e145738b5bafad1ecb1d565b52238e8bb93e626a1a918dfe0" exitCode=0 Sep 09 23:48:45.521456 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:45.521435 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cczv2" event={"ID":"e884e0c1-c063-4c91-a1d2-012bc3980235","Type":"ContainerDied","Data":"8a79b388102a865e145738b5bafad1ecb1d565b52238e8bb93e626a1a918dfe0"} Sep 09 23:48:46.233592 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:46.233430 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:46.233751 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:46.233663 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:46.526726 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:46.526697 2574 generic.go:356] "Generic (PLEG): container finished" podID="e884e0c1-c063-4c91-a1d2-012bc3980235" containerID="d033eb01118b8464f07a81f28e06685d8b2ea81de99b7d91d829ca09d4fa55ae" exitCode=0 Sep 09 23:48:46.527109 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:46.526740 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cczv2" event={"ID":"e884e0c1-c063-4c91-a1d2-012bc3980235","Type":"ContainerDied","Data":"d033eb01118b8464f07a81f28e06685d8b2ea81de99b7d91d829ca09d4fa55ae"} Sep 09 23:48:47.233740 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:47.233710 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:47.233740 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:47.233728 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:47.233966 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:47.233823 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:47.233966 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:47.233924 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:47.531802 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:47.531733 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cczv2" event={"ID":"e884e0c1-c063-4c91-a1d2-012bc3980235","Type":"ContainerStarted","Data":"66234ccf493facb5989344e4edfbfb37a3376e8d1447b9b7160c908b08977149"} Sep 09 23:48:47.553924 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:47.553880 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cczv2" podStartSLOduration=10.056378537 podStartE2EDuration="39.553866828s" totalImagesPullingTime="29.497488291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:47.553609024 +0000 UTC m=+39.760748794" watchObservedRunningTime="2026-09-09 23:48:47.553866828 +0000 UTC m=+39.761006599" Sep 09 23:48:48.234854 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:48.234820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:48.235021 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:48.234904 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:49.233360 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:49.233331 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:49.233360 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:49.233359 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:49.233834 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:49.233441 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:49.233834 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:49.233585 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:50.234070 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:50.234037 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:50.234468 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:50.234150 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:51.233177 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:51.233145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:51.233360 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:51.233145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:51.233360 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:51.233272 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:51.233360 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:51.233340 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:52.234012 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:52.233974 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:52.234406 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:52.234104 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:53.233559 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:53.233535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:53.233710 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:53.233534 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:53.233710 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:53.233646 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:53.233710 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:53.233699 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:54.233845 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:54.233809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:54.234202 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:54.233917 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:55.233681 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:55.233652 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:55.233859 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:55.233658 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:55.233859 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:55.233740 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:55.233859 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:55.233831 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:56.233818 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:56.233791 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:56.234002 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:56.233883 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjngb" podUID="55083d6e-c367-40eb-96ad-b70390fc1039" Sep 09 23:48:57.234031 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:57.234002 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:57.234449 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:57.234008 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:57.234449 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:57.234094 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mppg" podUID="dd26812f-6704-4784-98a0-a757cc61c378" Sep 09 23:48:57.234449 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:48:57.234181 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmqgc" podUID="eb88e26c-b785-40be-b781-088af7d86fe2" Sep 09 23:48:58.052054 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.052033 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-102.ec2.internal" event="NodeReady" Sep 09 23:48:58.052197 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.052121 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 23:48:58.084867 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.084836 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8648f96b6b-z5w6h"] Sep 09 23:48:58.086463 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.086445 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.088384 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.088364 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 23:48:58.088934 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.088916 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kmlm6\"" Sep 09 23:48:58.089013 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.088999 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 23:48:58.089264 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.089247 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 23:48:58.096357 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.096333 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 23:48:58.098806 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.098789 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4952r"] Sep 09 23:48:58.100348 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.100335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.102880 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.102851 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8648f96b6b-z5w6h"] Sep 09 23:48:58.103528 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.103505 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 23:48:58.103593 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.103539 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccffp\"" Sep 09 23:48:58.103785 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.103771 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 23:48:58.106393 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.106372 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fkgvh"] Sep 09 23:48:58.108203 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.108185 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fkgvh" Sep 09 23:48:58.110510 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.110492 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2qmv7\"" Sep 09 23:48:58.110510 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.110501 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 23:48:58.110649 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.110518 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 23:48:58.110649 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.110577 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 23:48:58.114748 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.114728 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4952r"] Sep 09 23:48:58.116556 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.116534 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fkgvh"] Sep 09 23:48:58.127201 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e147845-f357-4d4e-a183-c8a22727e954-metrics-tls\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.127311 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e147845-f357-4d4e-a183-c8a22727e954-tmp-dir\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.127311 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca8f41dc-9f5a-4bb4-9396-51490c020315-ca-trust-extracted\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.127311 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127266 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca8f41dc-9f5a-4bb4-9396-51490c020315-bound-sa-token\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.127444 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83f7ef84-a989-471a-ae8e-b12cad0c9279-cert\") pod \"ingress-canary-fkgvh\" (UID: \"83f7ef84-a989-471a-ae8e-b12cad0c9279\") " pod="openshift-ingress-canary/ingress-canary-fkgvh" Sep 09 23:48:58.127444 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvhjz\" (UniqueName: \"kubernetes.io/projected/3e147845-f357-4d4e-a183-c8a22727e954-kube-api-access-lvhjz\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.127444 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127381 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca8f41dc-9f5a-4bb4-9396-51490c020315-installation-pull-secrets\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.127444 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786k5\" (UniqueName: \"kubernetes.io/projected/ca8f41dc-9f5a-4bb4-9396-51490c020315-kube-api-access-786k5\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.127444 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca8f41dc-9f5a-4bb4-9396-51490c020315-registry-tls\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.127606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca8f41dc-9f5a-4bb4-9396-51490c020315-trusted-ca\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.127606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127502 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jrf\" (UniqueName: \"kubernetes.io/projected/83f7ef84-a989-471a-ae8e-b12cad0c9279-kube-api-access-d6jrf\") pod \"ingress-canary-fkgvh\" (UID: \"83f7ef84-a989-471a-ae8e-b12cad0c9279\") " pod="openshift-ingress-canary/ingress-canary-fkgvh" Sep 09 23:48:58.127606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca8f41dc-9f5a-4bb4-9396-51490c020315-registry-certificates\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.127606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e147845-f357-4d4e-a183-c8a22727e954-config-volume\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.127606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.127593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca8f41dc-9f5a-4bb4-9396-51490c020315-image-registry-private-configuration\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.228293 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e147845-f357-4d4e-a183-c8a22727e954-config-volume\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.228400 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca8f41dc-9f5a-4bb4-9396-51490c020315-image-registry-private-configuration\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.228400 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e147845-f357-4d4e-a183-c8a22727e954-metrics-tls\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.228400 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e147845-f357-4d4e-a183-c8a22727e954-tmp-dir\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.228400 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca8f41dc-9f5a-4bb4-9396-51490c020315-ca-trust-extracted\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.228601 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca8f41dc-9f5a-4bb4-9396-51490c020315-bound-sa-token\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.228601 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83f7ef84-a989-471a-ae8e-b12cad0c9279-cert\") pod \"ingress-canary-fkgvh\" (UID: \"83f7ef84-a989-471a-ae8e-b12cad0c9279\") " pod="openshift-ingress-canary/ingress-canary-fkgvh" Sep 09 23:48:58.228758 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvhjz\" (UniqueName: \"kubernetes.io/projected/3e147845-f357-4d4e-a183-c8a22727e954-kube-api-access-lvhjz\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.228878 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca8f41dc-9f5a-4bb4-9396-51490c020315-installation-pull-secrets\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.228878 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e147845-f357-4d4e-a183-c8a22727e954-tmp-dir\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.228878 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228796 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786k5\" (UniqueName: \"kubernetes.io/projected/ca8f41dc-9f5a-4bb4-9396-51490c020315-kube-api-access-786k5\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.228878 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca8f41dc-9f5a-4bb4-9396-51490c020315-registry-tls\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.228878 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca8f41dc-9f5a-4bb4-9396-51490c020315-trusted-ca\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.229126 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e147845-f357-4d4e-a183-c8a22727e954-config-volume\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.229126 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jrf\" (UniqueName: \"kubernetes.io/projected/83f7ef84-a989-471a-ae8e-b12cad0c9279-kube-api-access-d6jrf\") pod \"ingress-canary-fkgvh\" (UID: \"83f7ef84-a989-471a-ae8e-b12cad0c9279\") " pod="openshift-ingress-canary/ingress-canary-fkgvh" Sep 09 23:48:58.229126 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca8f41dc-9f5a-4bb4-9396-51490c020315-registry-certificates\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.229126 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.228973 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca8f41dc-9f5a-4bb4-9396-51490c020315-ca-trust-extracted\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.229884 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.229817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca8f41dc-9f5a-4bb4-9396-51490c020315-registry-certificates\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.229884 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.229830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca8f41dc-9f5a-4bb4-9396-51490c020315-trusted-ca\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.233247 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.232824 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca8f41dc-9f5a-4bb4-9396-51490c020315-image-registry-private-configuration\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.233247 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.232872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83f7ef84-a989-471a-ae8e-b12cad0c9279-cert\") pod \"ingress-canary-fkgvh\" (UID: \"83f7ef84-a989-471a-ae8e-b12cad0c9279\") " pod="openshift-ingress-canary/ingress-canary-fkgvh" Sep 09 23:48:58.233247 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.233152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e147845-f357-4d4e-a183-c8a22727e954-metrics-tls\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.234036 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.233892 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca8f41dc-9f5a-4bb4-9396-51490c020315-installation-pull-secrets\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.234129 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.234065 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca8f41dc-9f5a-4bb4-9396-51490c020315-registry-tls\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.234763 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.234745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:48:58.237121 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.237044 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 23:48:58.237467 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.237444 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca8f41dc-9f5a-4bb4-9396-51490c020315-bound-sa-token\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.238067 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.238041 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-786k5\" (UniqueName: \"kubernetes.io/projected/ca8f41dc-9f5a-4bb4-9396-51490c020315-kube-api-access-786k5\") pod \"image-registry-8648f96b6b-z5w6h\" (UID: \"ca8f41dc-9f5a-4bb4-9396-51490c020315\") " pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.238154 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.238095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvhjz\" (UniqueName: \"kubernetes.io/projected/3e147845-f357-4d4e-a183-c8a22727e954-kube-api-access-lvhjz\") pod \"dns-default-4952r\" (UID: \"3e147845-f357-4d4e-a183-c8a22727e954\") " pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.238491 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.238472 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jrf\" (UniqueName: \"kubernetes.io/projected/83f7ef84-a989-471a-ae8e-b12cad0c9279-kube-api-access-d6jrf\") pod \"ingress-canary-fkgvh\" (UID: \"83f7ef84-a989-471a-ae8e-b12cad0c9279\") " pod="openshift-ingress-canary/ingress-canary-fkgvh" Sep 09 23:48:58.397866 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.397804 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:58.409434 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.409412 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4952r" Sep 09 23:48:58.416574 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.416556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fkgvh" Sep 09 23:48:58.678652 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.678582 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4952r"] Sep 09 23:48:58.681294 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.681149 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8648f96b6b-z5w6h"] Sep 09 23:48:58.682092 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:58.682065 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e147845_f357_4d4e_a183_c8a22727e954.slice/crio-df38ca4b70299f44d27cff558f332fdf57cbee9c3ef0bbf9e1519d0f14acd063 WatchSource:0}: Error finding container df38ca4b70299f44d27cff558f332fdf57cbee9c3ef0bbf9e1519d0f14acd063: Status 404 returned error can't find the container with id df38ca4b70299f44d27cff558f332fdf57cbee9c3ef0bbf9e1519d0f14acd063 Sep 09 23:48:58.685390 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:58.685367 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8f41dc_9f5a_4bb4_9396_51490c020315.slice/crio-231f0cb874e9721df0be423befaa234d22bdc0a5a173a537fdce95566ca786a6 WatchSource:0}: Error finding container 231f0cb874e9721df0be423befaa234d22bdc0a5a173a537fdce95566ca786a6: Status 404 returned error can't find the container with id 231f0cb874e9721df0be423befaa234d22bdc0a5a173a537fdce95566ca786a6 Sep 09 23:48:58.698643 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:58.698619 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fkgvh"] Sep 09 23:48:58.703031 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:48:58.703004 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f7ef84_a989_471a_ae8e_b12cad0c9279.slice/crio-eb0148332b4efdbef048d83e88a674c2aeae46c6f06845fe4ac8b81aeef6a3b0 WatchSource:0}: Error finding container eb0148332b4efdbef048d83e88a674c2aeae46c6f06845fe4ac8b81aeef6a3b0: Status 404 returned error can't find the container with id eb0148332b4efdbef048d83e88a674c2aeae46c6f06845fe4ac8b81aeef6a3b0 Sep 09 23:48:59.233602 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.233320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:48:59.233602 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.233320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:48:59.235937 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.235897 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 23:48:59.237183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.236894 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 23:48:59.237183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.236911 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s7wg5\"" Sep 09 23:48:59.237183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.236914 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 23:48:59.237183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.236960 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n7tkk\"" Sep 09 23:48:59.564725 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.564686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fkgvh" event={"ID":"83f7ef84-a989-471a-ae8e-b12cad0c9279","Type":"ContainerStarted","Data":"eb0148332b4efdbef048d83e88a674c2aeae46c6f06845fe4ac8b81aeef6a3b0"} Sep 09 23:48:59.566164 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.566134 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4952r" event={"ID":"3e147845-f357-4d4e-a183-c8a22727e954","Type":"ContainerStarted","Data":"df38ca4b70299f44d27cff558f332fdf57cbee9c3ef0bbf9e1519d0f14acd063"} Sep 09 23:48:59.568899 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.568869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" event={"ID":"ca8f41dc-9f5a-4bb4-9396-51490c020315","Type":"ContainerStarted","Data":"6819c86491578eb8b2fe71bbd253c34ad457f23adc6d5c5eef30271253fbfbc1"} Sep 09 23:48:59.568978 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.568920 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" event={"ID":"ca8f41dc-9f5a-4bb4-9396-51490c020315","Type":"ContainerStarted","Data":"231f0cb874e9721df0be423befaa234d22bdc0a5a173a537fdce95566ca786a6"} Sep 09 23:48:59.569049 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.569036 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:48:59.587485 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:48:59.587435 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" podStartSLOduration=2.587417682 podStartE2EDuration="2.587417682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:59.58688286 +0000 UTC m=+51.794022614" watchObservedRunningTime="2026-09-09 23:48:59.587417682 +0000 UTC m=+51.794557450" Sep 09 23:49:01.577755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:01.577712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fkgvh" event={"ID":"83f7ef84-a989-471a-ae8e-b12cad0c9279","Type":"ContainerStarted","Data":"68ce96da075cd52c596b96301c2fc3839d48d83fe56587f4e5a117099fb9d70f"} Sep 09 23:49:01.579613 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:01.579585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4952r" event={"ID":"3e147845-f357-4d4e-a183-c8a22727e954","Type":"ContainerStarted","Data":"20cb3cfd4b49da5d1a6139c72cf52e1ca28159120c8f0b48695a3f82e8dfdf88"} Sep 09 23:49:01.579750 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:01.579633 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4952r" event={"ID":"3e147845-f357-4d4e-a183-c8a22727e954","Type":"ContainerStarted","Data":"3808ead23d8e94d4b151cd2fb55081bc8c6048c4e6576a9e9b181ca4cce29e9c"} Sep 09 23:49:01.579750 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:01.579662 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4952r" Sep 09 23:49:01.591778 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:01.591731 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fkgvh" podStartSLOduration=1.211469495 podStartE2EDuration="3.591713571s" totalImagesPullingTime="2.380244076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:01.590735312 +0000 UTC m=+53.797875099" watchObservedRunningTime="2026-09-09 23:49:01.591713571 +0000 UTC m=+53.798853339" Sep 09 23:49:01.605640 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:01.605568 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4952r" podStartSLOduration=1.207550569 podStartE2EDuration="3.605549725s" totalImagesPullingTime="2.397999156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:01.605323124 +0000 UTC m=+53.812462893" watchObservedRunningTime="2026-09-09 23:49:01.605549725 +0000 UTC m=+53.812689495" Sep 09 23:49:03.201836 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.201806 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kxtp9"] Sep 09 23:49:03.203990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.203965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.207448 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.207424 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 23:49:03.207448 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.207420 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 23:49:03.207627 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.207518 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 23:49:03.207680 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.207666 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 23:49:03.208460 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.208433 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rjj8g\"" Sep 09 23:49:03.217298 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.217256 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kxtp9"] Sep 09 23:49:03.266222 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.266196 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-crio-socket\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.266358 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.266242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9vc\" (UniqueName: \"kubernetes.io/projected/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-api-access-sg9vc\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.266358 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.266271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-data-volume\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.266358 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.266299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.266358 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.266322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.293316 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.293268 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-jfvwq"] Sep 09 23:49:03.295137 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.295120 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-jfvwq" Sep 09 23:49:03.297426 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.297409 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 23:49:03.297513 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.297471 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 23:49:03.297513 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.297481 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lqwzv\"" Sep 09 23:49:03.304988 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.304967 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-jfvwq"] Sep 09 23:49:03.366751 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.366728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftkqn\" (UniqueName: \"kubernetes.io/projected/454960f3-186d-4d90-8801-ff88241088b9-kube-api-access-ftkqn\") pod \"downloads-85d98c998f-jfvwq\" (UID: \"454960f3-186d-4d90-8801-ff88241088b9\") " pod="openshift-console/downloads-85d98c998f-jfvwq" Sep 09 23:49:03.366868 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.366769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-crio-socket\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.366868 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.366835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9vc\" (UniqueName: \"kubernetes.io/projected/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-api-access-sg9vc\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.366990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.366887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-data-volume\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.366990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.366905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.366990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.366920 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.366990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.366951 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-crio-socket\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.367178 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:03.367028 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:49:03.367178 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:03.367039 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:49:03.367178 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:03.367088 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls podName:07f5c089-1b0e-4b58-b03b-8c1a586eaf28 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:03.867071755 +0000 UTC m=+56.074211520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kxtp9" (UID: "07f5c089-1b0e-4b58-b03b-8c1a586eaf28") : secret "insights-runtime-extractor-tls" not found Sep 09 23:49:03.367178 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:03.367110 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm podName:07f5c089-1b0e-4b58-b03b-8c1a586eaf28 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:03.867101553 +0000 UTC m=+56.074241352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kxtp9" (UID: "07f5c089-1b0e-4b58-b03b-8c1a586eaf28") : configmap "kube-rbac-proxy" not found Sep 09 23:49:03.367360 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.367234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-data-volume\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.379195 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.379169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9vc\" (UniqueName: \"kubernetes.io/projected/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-api-access-sg9vc\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.468016 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.467961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftkqn\" (UniqueName: \"kubernetes.io/projected/454960f3-186d-4d90-8801-ff88241088b9-kube-api-access-ftkqn\") pod \"downloads-85d98c998f-jfvwq\" (UID: \"454960f3-186d-4d90-8801-ff88241088b9\") " pod="openshift-console/downloads-85d98c998f-jfvwq" Sep 09 23:49:03.475378 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.475357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftkqn\" (UniqueName: \"kubernetes.io/projected/454960f3-186d-4d90-8801-ff88241088b9-kube-api-access-ftkqn\") pod \"downloads-85d98c998f-jfvwq\" (UID: \"454960f3-186d-4d90-8801-ff88241088b9\") " pod="openshift-console/downloads-85d98c998f-jfvwq" Sep 09 23:49:03.604225 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.604200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-jfvwq" Sep 09 23:49:03.815221 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.815197 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4952r_3e147845-f357-4d4e-a183-c8a22727e954/dns/0.log" Sep 09 23:49:03.870441 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.870410 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-jfvwq"] Sep 09 23:49:03.873442 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:03.873419 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454960f3_186d_4d90_8801_ff88241088b9.slice/crio-20ee9de3a78763d44aa5b048a9a4a840ccd75d6e218e6f14d45d5351081e4daf WatchSource:0}: Error finding container 20ee9de3a78763d44aa5b048a9a4a840ccd75d6e218e6f14d45d5351081e4daf: Status 404 returned error can't find the container with id 20ee9de3a78763d44aa5b048a9a4a840ccd75d6e218e6f14d45d5351081e4daf Sep 09 23:49:03.877853 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.877830 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.877918 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.877867 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:03.877960 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:03.877921 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:49:03.877994 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:03.877978 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm podName:07f5c089-1b0e-4b58-b03b-8c1a586eaf28 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:04.877961074 +0000 UTC m=+57.085100840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kxtp9" (UID: "07f5c089-1b0e-4b58-b03b-8c1a586eaf28") : configmap "kube-rbac-proxy" not found Sep 09 23:49:03.877994 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:03.877986 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:49:03.878077 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:03.878024 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls podName:07f5c089-1b0e-4b58-b03b-8c1a586eaf28 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:04.878016566 +0000 UTC m=+57.085156326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kxtp9" (UID: "07f5c089-1b0e-4b58-b03b-8c1a586eaf28") : secret "insights-runtime-extractor-tls" not found Sep 09 23:49:03.999315 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:03.999273 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4952r_3e147845-f357-4d4e-a183-c8a22727e954/kube-rbac-proxy/0.log" Sep 09 23:49:04.588391 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:04.588343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-jfvwq" event={"ID":"454960f3-186d-4d90-8801-ff88241088b9","Type":"ContainerStarted","Data":"20ee9de3a78763d44aa5b048a9a4a840ccd75d6e218e6f14d45d5351081e4daf"} Sep 09 23:49:04.885316 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:04.885213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:04.885316 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:04.885256 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:04.885532 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:04.885370 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:49:04.885532 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:04.885406 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:49:04.885532 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:04.885444 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm podName:07f5c089-1b0e-4b58-b03b-8c1a586eaf28 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:06.885421965 +0000 UTC m=+59.092561737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kxtp9" (UID: "07f5c089-1b0e-4b58-b03b-8c1a586eaf28") : configmap "kube-rbac-proxy" not found Sep 09 23:49:04.885532 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:04.885461 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls podName:07f5c089-1b0e-4b58-b03b-8c1a586eaf28 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:06.885454944 +0000 UTC m=+59.092594691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kxtp9" (UID: "07f5c089-1b0e-4b58-b03b-8c1a586eaf28") : secret "insights-runtime-extractor-tls" not found Sep 09 23:49:05.398819 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:05.398785 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dmks5_59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af/dns-node-resolver/0.log" Sep 09 23:49:05.809606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:05.809576 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8648f96b6b-z5w6h_ca8f41dc-9f5a-4bb4-9396-51490c020315/registry/0.log" Sep 09 23:49:05.999429 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:05.999396 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5ksjl_44d663d3-f7dc-4270-94cc-696d3b55ea0c/node-ca/0.log" Sep 09 23:49:06.902322 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:06.902271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:06.902761 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:06.902337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:06.902881 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:06.902848 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:06.904821 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:06.904800 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07f5c089-1b0e-4b58-b03b-8c1a586eaf28-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxtp9\" (UID: \"07f5c089-1b0e-4b58-b03b-8c1a586eaf28\") " pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:07.112097 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:07.112063 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kxtp9" Sep 09 23:49:07.199739 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:07.199713 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fkgvh_83f7ef84-a989-471a-ae8e-b12cad0c9279/serve-healthcheck-canary/0.log" Sep 09 23:49:07.390823 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:07.390773 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f5c089_1b0e_4b58_b03b_8c1a586eaf28.slice/crio-d99fc8e3447403cbbeae15eed19c30c4fe499fb5918d447e727c472990fe55a4 WatchSource:0}: Error finding container d99fc8e3447403cbbeae15eed19c30c4fe499fb5918d447e727c472990fe55a4: Status 404 returned error can't find the container with id d99fc8e3447403cbbeae15eed19c30c4fe499fb5918d447e727c472990fe55a4 Sep 09 23:49:07.391806 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:07.391773 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kxtp9"] Sep 09 23:49:07.599251 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:07.599210 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kxtp9" event={"ID":"07f5c089-1b0e-4b58-b03b-8c1a586eaf28","Type":"ContainerStarted","Data":"03c96f21f0e033ba7aa9f1345f296848e6bce74abe23a821f7c1c46eb8ec30ed"} Sep 09 23:49:07.599406 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:07.599306 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kxtp9" event={"ID":"07f5c089-1b0e-4b58-b03b-8c1a586eaf28","Type":"ContainerStarted","Data":"d99fc8e3447403cbbeae15eed19c30c4fe499fb5918d447e727c472990fe55a4"} Sep 09 23:49:09.564020 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:09.563979 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwxwt" Sep 09 23:49:09.612974 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:09.612923 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kxtp9" event={"ID":"07f5c089-1b0e-4b58-b03b-8c1a586eaf28","Type":"ContainerStarted","Data":"efce1fe510bd995b069ebd41b4e9edbe3aa4f6f539a93009e091c14668db2311"} Sep 09 23:49:10.618492 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:10.618418 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kxtp9" event={"ID":"07f5c089-1b0e-4b58-b03b-8c1a586eaf28","Type":"ContainerStarted","Data":"87c5a79743c86f172f7edf6fb806240486fdffcda56478c49111633ae52ea6e1"} Sep 09 23:49:10.638399 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:10.638338 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kxtp9" podStartSLOduration=5.231949935 podStartE2EDuration="7.638322159s" totalImagesPullingTime="2.406372224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:49:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:10.636923198 +0000 UTC m=+62.844062969" watchObservedRunningTime="2026-09-09 23:49:10.638322159 +0000 UTC m=+62.845461928" Sep 09 23:49:11.585184 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:11.585155 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4952r" Sep 09 23:49:13.372671 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.372636 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6"] Sep 09 23:49:13.376881 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.376855 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.379913 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.379882 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 23:49:13.380036 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.379924 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 23:49:13.380036 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.379935 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 09 23:49:13.380371 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.380195 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 09 23:49:13.380371 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.380208 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rxtdj\"" Sep 09 23:49:13.380371 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.380247 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 23:49:13.387395 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.387374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6"] Sep 09 23:49:13.397616 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.397594 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f6tjk"] Sep 09 23:49:13.400585 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.400566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.402568 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.402542 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 23:49:13.402666 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.402567 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zrbrm\"" Sep 09 23:49:13.403063 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.402987 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 23:49:13.404561 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.403909 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 23:49:13.454110 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7061de57-0809-471e-9535-548e8218cce4-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.454307 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454118 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-root\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.454307 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7061de57-0809-471e-9535-548e8218cce4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.454307 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-metrics-client-ca\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.454307 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-tls\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.454516 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.454516 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454465 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-sys\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.454516 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454497 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-textfile\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.454663 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tnvx\" (UniqueName: \"kubernetes.io/projected/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-kube-api-access-7tnvx\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.454663 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7061de57-0809-471e-9535-548e8218cce4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.454663 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.454797 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-wtmp\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.454797 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.454696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrm7l\" (UniqueName: \"kubernetes.io/projected/7061de57-0809-471e-9535-548e8218cce4-kube-api-access-mrm7l\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.555087 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-wtmp\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555087 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrm7l\" (UniqueName: \"kubernetes.io/projected/7061de57-0809-471e-9535-548e8218cce4-kube-api-access-mrm7l\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.555424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7061de57-0809-471e-9535-548e8218cce4-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.555424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-root\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7061de57-0809-471e-9535-548e8218cce4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.555424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-metrics-client-ca\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-tls\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555265 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-wtmp\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-sys\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555424 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-textfile\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555879 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tnvx\" (UniqueName: \"kubernetes.io/projected/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-kube-api-access-7tnvx\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555879 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7061de57-0809-471e-9535-548e8218cce4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.555879 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555879 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555557 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-root\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.555879 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-sys\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.556131 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555889 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.556131 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.555985 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7061de57-0809-471e-9535-548e8218cce4-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.556131 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.556039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-textfile\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.556131 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:13.556062 2574 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 09 23:49:13.556131 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:13.556123 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7061de57-0809-471e-9535-548e8218cce4-openshift-state-metrics-tls podName:7061de57-0809-471e-9535-548e8218cce4 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:14.056104857 +0000 UTC m=+66.263244619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7061de57-0809-471e-9535-548e8218cce4-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-gc7z6" (UID: "7061de57-0809-471e-9535-548e8218cce4") : secret "openshift-state-metrics-tls" not found Sep 09 23:49:13.556416 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.556376 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-metrics-client-ca\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.557942 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.557910 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.558366 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.558345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7061de57-0809-471e-9535-548e8218cce4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.558927 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.558902 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-node-exporter-tls\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.566357 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.566334 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrm7l\" (UniqueName: \"kubernetes.io/projected/7061de57-0809-471e-9535-548e8218cce4-kube-api-access-mrm7l\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:13.566576 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.566559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tnvx\" (UniqueName: \"kubernetes.io/projected/c77e3ae3-38bf-4e62-9a5c-c690eef779d7-kube-api-access-7tnvx\") pod \"node-exporter-f6tjk\" (UID: \"c77e3ae3-38bf-4e62-9a5c-c690eef779d7\") " pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.710799 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.710717 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f6tjk" Sep 09 23:49:13.958146 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.958104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:49:13.960675 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.960643 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 23:49:13.971713 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:13.971656 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb88e26c-b785-40be-b781-088af7d86fe2-metrics-certs\") pod \"network-metrics-daemon-gmqgc\" (UID: \"eb88e26c-b785-40be-b781-088af7d86fe2\") " pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:49:14.059174 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.059133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7061de57-0809-471e-9535-548e8218cce4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:14.059353 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.059204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:49:14.061564 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.061541 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 23:49:14.061734 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.061709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7061de57-0809-471e-9535-548e8218cce4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gc7z6\" (UID: \"7061de57-0809-471e-9535-548e8218cce4\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:14.072158 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.072134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55083d6e-c367-40eb-96ad-b70390fc1039-original-pull-secret\") pod \"global-pull-secret-syncer-bjngb\" (UID: \"55083d6e-c367-40eb-96ad-b70390fc1039\") " pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:49:14.154557 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.154522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjngb" Sep 09 23:49:14.159756 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.159733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:49:14.161815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.161795 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 23:49:14.172640 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.172615 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 23:49:14.183800 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.183777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn2sx\" (UniqueName: \"kubernetes.io/projected/dd26812f-6704-4784-98a0-a757cc61c378-kube-api-access-cn2sx\") pod \"network-check-target-6mppg\" (UID: \"dd26812f-6704-4784-98a0-a757cc61c378\") " pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:49:14.256901 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.256827 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n7tkk\"" Sep 09 23:49:14.263375 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.263349 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s7wg5\"" Sep 09 23:49:14.264960 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.264945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmqgc" Sep 09 23:49:14.272640 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.272624 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:49:14.288206 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.288187 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" Sep 09 23:49:14.288836 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.288807 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f9646598-ttcf5"] Sep 09 23:49:14.291696 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.291678 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.294133 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.294101 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 23:49:14.294235 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.294199 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 23:49:14.294322 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.294239 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 23:49:14.294384 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.294335 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rtsgv\"" Sep 09 23:49:14.294384 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.294358 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 23:49:14.294914 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.294897 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 23:49:14.300933 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.300888 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f9646598-ttcf5"] Sep 09 23:49:14.361118 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.361090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-oauth-config\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.361312 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.361132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-serving-cert\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.361312 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.361201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-oauth-serving-cert\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.361312 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.361260 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2brg\" (UniqueName: \"kubernetes.io/projected/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-kube-api-access-h2brg\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.361312 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.361303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-config\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.361542 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.361325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-service-ca\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.461699 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.461664 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2brg\" (UniqueName: \"kubernetes.io/projected/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-kube-api-access-h2brg\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.461699 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.461700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-config\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.462193 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.461719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-service-ca\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.462193 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.461751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-oauth-config\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.462193 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.461791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-serving-cert\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.462193 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.461832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-oauth-serving-cert\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.462551 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.462525 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-service-ca\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.462678 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.462569 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-oauth-serving-cert\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.462678 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.462573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-config\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.464470 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.464452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-oauth-config\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.464665 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.464646 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-serving-cert\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.468752 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.468731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2brg\" (UniqueName: \"kubernetes.io/projected/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-kube-api-access-h2brg\") pod \"console-5f9646598-ttcf5\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:14.602707 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:14.602663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:17.720807 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.720773 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-587c665f98-fhqhx"] Sep 09 23:49:17.722935 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.722913 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.725264 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.725241 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 09 23:49:17.726157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.726112 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 23:49:17.726350 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.726249 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 09 23:49:17.726431 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.726409 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2p0iatijkluq3\"" Sep 09 23:49:17.726507 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.726481 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5md2r\"" Sep 09 23:49:17.726594 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.726577 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 09 23:49:17.740537 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.740514 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-587c665f98-fhqhx"] Sep 09 23:49:17.786774 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.786742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hrtv\" (UniqueName: \"kubernetes.io/projected/a9a255bc-472a-46da-9df5-2a0277aa6369-kube-api-access-8hrtv\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.786931 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.786794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a9a255bc-472a-46da-9df5-2a0277aa6369-secret-metrics-server-tls\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.786931 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.786896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9a255bc-472a-46da-9df5-2a0277aa6369-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.787051 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.786945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a9a255bc-472a-46da-9df5-2a0277aa6369-audit-log\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.787051 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.786965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a9a255bc-472a-46da-9df5-2a0277aa6369-secret-metrics-server-client-certs\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.787051 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.786993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a9a255bc-472a-46da-9df5-2a0277aa6369-client-ca-bundle\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.787051 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.787023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a9a255bc-472a-46da-9df5-2a0277aa6369-metrics-server-audit-profiles\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.887785 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.887745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9a255bc-472a-46da-9df5-2a0277aa6369-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.887935 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.887882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a9a255bc-472a-46da-9df5-2a0277aa6369-audit-log\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.887935 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.887916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a9a255bc-472a-46da-9df5-2a0277aa6369-secret-metrics-server-client-certs\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.888043 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.887947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a9a255bc-472a-46da-9df5-2a0277aa6369-client-ca-bundle\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.888043 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.887976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a9a255bc-472a-46da-9df5-2a0277aa6369-metrics-server-audit-profiles\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.888146 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.888045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrtv\" (UniqueName: \"kubernetes.io/projected/a9a255bc-472a-46da-9df5-2a0277aa6369-kube-api-access-8hrtv\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.888146 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.888078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a9a255bc-472a-46da-9df5-2a0277aa6369-secret-metrics-server-tls\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.888680 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.888649 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9a255bc-472a-46da-9df5-2a0277aa6369-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.888835 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.888806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a9a255bc-472a-46da-9df5-2a0277aa6369-audit-log\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.889043 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.889012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a9a255bc-472a-46da-9df5-2a0277aa6369-metrics-server-audit-profiles\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.890821 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.890795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a9a255bc-472a-46da-9df5-2a0277aa6369-client-ca-bundle\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.890920 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.890809 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a9a255bc-472a-46da-9df5-2a0277aa6369-secret-metrics-server-tls\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.890985 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.890911 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a9a255bc-472a-46da-9df5-2a0277aa6369-secret-metrics-server-client-certs\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:17.895557 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:17.895512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrtv\" (UniqueName: \"kubernetes.io/projected/a9a255bc-472a-46da-9df5-2a0277aa6369-kube-api-access-8hrtv\") pod \"metrics-server-587c665f98-fhqhx\" (UID: \"a9a255bc-472a-46da-9df5-2a0277aa6369\") " pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:18.034665 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.034585 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:18.161959 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.161927 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp"] Sep 09 23:49:18.165163 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.165134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:18.167919 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.167895 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k72d6\"" Sep 09 23:49:18.167919 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.167904 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 09 23:49:18.171417 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.171396 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp"] Sep 09 23:49:18.190889 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.190862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-kxbpp\" (UID: \"4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:18.191011 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.190926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdbr\" (UniqueName: \"kubernetes.io/projected/4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6-kube-api-access-2jdbr\") pod \"monitoring-plugin-85bbbbbf4-kxbpp\" (UID: \"4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:18.291660 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.291628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdbr\" (UniqueName: \"kubernetes.io/projected/4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6-kube-api-access-2jdbr\") pod \"monitoring-plugin-85bbbbbf4-kxbpp\" (UID: \"4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:18.291828 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.291705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-kxbpp\" (UID: \"4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:18.291828 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:18.291799 2574 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 09 23:49:18.291941 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:18.291863 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6-monitoring-plugin-cert podName:4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:18.791844189 +0000 UTC m=+70.998983950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-kxbpp" (UID: "4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6") : secret "monitoring-plugin-cert" not found Sep 09 23:49:18.299847 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.299822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdbr\" (UniqueName: \"kubernetes.io/projected/4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6-kube-api-access-2jdbr\") pod \"monitoring-plugin-85bbbbbf4-kxbpp\" (UID: \"4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:18.593400 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.593304 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6b5787994-8f97s"] Sep 09 23:49:18.595813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.595795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.598141 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.598117 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 23:49:18.598141 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.598143 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 23:49:18.598410 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.598185 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 23:49:18.598410 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.598117 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 23:49:18.598410 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.598117 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 23:49:18.598818 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.598801 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tm8rq\"" Sep 09 23:49:18.602978 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.602952 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 23:49:18.607304 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.607264 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b5787994-8f97s"] Sep 09 23:49:18.694740 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.694708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-secret-telemeter-client\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.694901 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.694791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rvj\" (UniqueName: \"kubernetes.io/projected/4305d5fc-da8f-4379-805c-b56c2fb77df1-kube-api-access-42rvj\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.694901 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.694825 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-federate-client-tls\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.694901 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.694860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.695041 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.694901 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4305d5fc-da8f-4379-805c-b56c2fb77df1-serving-certs-ca-bundle\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.695041 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.694929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4305d5fc-da8f-4379-805c-b56c2fb77df1-metrics-client-ca\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.695041 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.694964 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-telemeter-client-tls\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.695041 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.695016 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4305d5fc-da8f-4379-805c-b56c2fb77df1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.796163 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.796128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-kxbpp\" (UID: \"4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:18.796625 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.796184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rvj\" (UniqueName: \"kubernetes.io/projected/4305d5fc-da8f-4379-805c-b56c2fb77df1-kube-api-access-42rvj\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.796625 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.796479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-federate-client-tls\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.796625 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.796526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.796625 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.796558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4305d5fc-da8f-4379-805c-b56c2fb77df1-serving-certs-ca-bundle\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.796625 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.796585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4305d5fc-da8f-4379-805c-b56c2fb77df1-metrics-client-ca\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.796625 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.796617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-telemeter-client-tls\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.796921 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.796647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4305d5fc-da8f-4379-805c-b56c2fb77df1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.796921 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.796703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-secret-telemeter-client\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.797609 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.797547 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4305d5fc-da8f-4379-805c-b56c2fb77df1-metrics-client-ca\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.797731 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.797707 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4305d5fc-da8f-4379-805c-b56c2fb77df1-serving-certs-ca-bundle\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.797849 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.797820 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4305d5fc-da8f-4379-805c-b56c2fb77df1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.799276 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.799258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-kxbpp\" (UID: \"4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:18.799760 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.799738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-secret-telemeter-client\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.799843 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.799783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-federate-client-tls\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.800172 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.800150 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-telemeter-client-tls\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.800750 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.800729 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4305d5fc-da8f-4379-805c-b56c2fb77df1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.803801 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.803782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rvj\" (UniqueName: \"kubernetes.io/projected/4305d5fc-da8f-4379-805c-b56c2fb77df1-kube-api-access-42rvj\") pod \"telemeter-client-6b5787994-8f97s\" (UID: \"4305d5fc-da8f-4379-805c-b56c2fb77df1\") " pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:18.906382 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:18.906313 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" Sep 09 23:49:19.076719 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.076682 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:19.632604 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:19.632545 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77e3ae3_38bf_4e62_9a5c_c690eef779d7.slice/crio-9914c423eaea6f82f9f752b4b7dd22649a2b4a8fa47f6f69c33453fa77d94fee WatchSource:0}: Error finding container 9914c423eaea6f82f9f752b4b7dd22649a2b4a8fa47f6f69c33453fa77d94fee: Status 404 returned error can't find the container with id 9914c423eaea6f82f9f752b4b7dd22649a2b4a8fa47f6f69c33453fa77d94fee Sep 09 23:49:19.656496 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.656416 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f6tjk" event={"ID":"c77e3ae3-38bf-4e62-9a5c-c690eef779d7","Type":"ContainerStarted","Data":"9914c423eaea6f82f9f752b4b7dd22649a2b4a8fa47f6f69c33453fa77d94fee"} Sep 09 23:49:19.886697 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.886605 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6mppg"] Sep 09 23:49:19.889812 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:19.889785 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd26812f_6704_4784_98a0_a757cc61c378.slice/crio-c9e4f69632c8ddab0878ffd786ca6bdf28d8fc742b6ca6a662e18ded9d070688 WatchSource:0}: Error finding container c9e4f69632c8ddab0878ffd786ca6bdf28d8fc742b6ca6a662e18ded9d070688: Status 404 returned error can't find the container with id c9e4f69632c8ddab0878ffd786ca6bdf28d8fc742b6ca6a662e18ded9d070688 Sep 09 23:49:19.920990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.920959 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f9646598-ttcf5"] Sep 09 23:49:19.923805 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:19.923777 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950d9a0c_3687_4a5a_8d76_6b2b0d4dd19f.slice/crio-40bd6aeaea6609f32d0c08c1c523e3b774d573cfc46902ed4a7852b691207828 WatchSource:0}: Error finding container 40bd6aeaea6609f32d0c08c1c523e3b774d573cfc46902ed4a7852b691207828: Status 404 returned error can't find the container with id 40bd6aeaea6609f32d0c08c1c523e3b774d573cfc46902ed4a7852b691207828 Sep 09 23:49:19.948927 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.948856 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-587c665f98-fhqhx"] Sep 09 23:49:19.953673 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:19.953643 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a255bc_472a_46da_9df5_2a0277aa6369.slice/crio-ffe52801e81cc760894b543fa68418d21aa693f3af6d9380b29cc4001c0c2326 WatchSource:0}: Error finding container ffe52801e81cc760894b543fa68418d21aa693f3af6d9380b29cc4001c0c2326: Status 404 returned error can't find the container with id ffe52801e81cc760894b543fa68418d21aa693f3af6d9380b29cc4001c0c2326 Sep 09 23:49:19.954438 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.954391 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b5787994-8f97s"] Sep 09 23:49:19.962722 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:19.962688 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4305d5fc_da8f_4379_805c_b56c2fb77df1.slice/crio-50e9ed6614e856126749a41fc45fb14c6459871bff03d72cbbc47d387ac5927e WatchSource:0}: Error finding container 50e9ed6614e856126749a41fc45fb14c6459871bff03d72cbbc47d387ac5927e: Status 404 returned error can't find the container with id 50e9ed6614e856126749a41fc45fb14c6459871bff03d72cbbc47d387ac5927e Sep 09 23:49:19.969171 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.969015 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjngb"] Sep 09 23:49:19.969171 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.969099 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp"] Sep 09 23:49:19.972742 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.972720 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6"] Sep 09 23:49:19.976513 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:19.976252 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7061de57_0809_471e_9535_548e8218cce4.slice/crio-f0fde37dcfaa9f1834f3990419b8ae05a53d9a1a3386027875d8dc4c5899d884 WatchSource:0}: Error finding container f0fde37dcfaa9f1834f3990419b8ae05a53d9a1a3386027875d8dc4c5899d884: Status 404 returned error can't find the container with id f0fde37dcfaa9f1834f3990419b8ae05a53d9a1a3386027875d8dc4c5899d884 Sep 09 23:49:19.985684 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:19.985637 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gmqgc"] Sep 09 23:49:19.989153 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:19.989126 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb88e26c_b785_40be_b781_088af7d86fe2.slice/crio-86f43de42d2b2e8179a69e7f7319c9227ca06984722b030954cbc4607fa0ead5 WatchSource:0}: Error finding container 86f43de42d2b2e8179a69e7f7319c9227ca06984722b030954cbc4607fa0ead5: Status 404 returned error can't find the container with id 86f43de42d2b2e8179a69e7f7319c9227ca06984722b030954cbc4607fa0ead5 Sep 09 23:49:20.579577 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.578721 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8648f96b6b-z5w6h" Sep 09 23:49:20.664979 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.664882 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gmqgc" event={"ID":"eb88e26c-b785-40be-b781-088af7d86fe2","Type":"ContainerStarted","Data":"86f43de42d2b2e8179a69e7f7319c9227ca06984722b030954cbc4607fa0ead5"} Sep 09 23:49:20.666984 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.666908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" event={"ID":"4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6","Type":"ContainerStarted","Data":"dcf602687b3ab495cc0e8dbff61e6a1c953580b5920eb8bb39f2b4870bd3988b"} Sep 09 23:49:20.668750 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.668671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" event={"ID":"4305d5fc-da8f-4379-805c-b56c2fb77df1","Type":"ContainerStarted","Data":"50e9ed6614e856126749a41fc45fb14c6459871bff03d72cbbc47d387ac5927e"} Sep 09 23:49:20.671471 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.671404 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjngb" event={"ID":"55083d6e-c367-40eb-96ad-b70390fc1039","Type":"ContainerStarted","Data":"a7c597a0a9287cea8501ab63e7b6936fefa197fa9892384fe75a7b5f6132a381"} Sep 09 23:49:20.677440 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.674938 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-jfvwq" event={"ID":"454960f3-186d-4d90-8801-ff88241088b9","Type":"ContainerStarted","Data":"26a93939a62ae88dfbcec3796e7dcd758221fcc9fa4f11c88705bc0fe28c1d6f"} Sep 09 23:49:20.677440 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.675901 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-jfvwq" Sep 09 23:49:20.681974 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.681940 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-jfvwq" Sep 09 23:49:20.682407 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.682381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" event={"ID":"7061de57-0809-471e-9535-548e8218cce4","Type":"ContainerStarted","Data":"6f1e55f0a59f0dc6f75fc1b92a96688fe9840ed2042c6095f005be04f07a9ab9"} Sep 09 23:49:20.682506 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.682436 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" event={"ID":"7061de57-0809-471e-9535-548e8218cce4","Type":"ContainerStarted","Data":"c7a4da455999ceb04445832677ffc90fe416347b9fa5685cb38502a0ee42f616"} Sep 09 23:49:20.682506 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.682468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" event={"ID":"7061de57-0809-471e-9535-548e8218cce4","Type":"ContainerStarted","Data":"f0fde37dcfaa9f1834f3990419b8ae05a53d9a1a3386027875d8dc4c5899d884"} Sep 09 23:49:20.685748 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.685686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" event={"ID":"a9a255bc-472a-46da-9df5-2a0277aa6369","Type":"ContainerStarted","Data":"ffe52801e81cc760894b543fa68418d21aa693f3af6d9380b29cc4001c0c2326"} Sep 09 23:49:20.687584 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.687527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6mppg" event={"ID":"dd26812f-6704-4784-98a0-a757cc61c378","Type":"ContainerStarted","Data":"c9e4f69632c8ddab0878ffd786ca6bdf28d8fc742b6ca6a662e18ded9d070688"} Sep 09 23:49:20.689189 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.689140 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f9646598-ttcf5" event={"ID":"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f","Type":"ContainerStarted","Data":"40bd6aeaea6609f32d0c08c1c523e3b774d573cfc46902ed4a7852b691207828"} Sep 09 23:49:20.691694 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:20.691652 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-jfvwq" podStartSLOduration=1.854231803 podStartE2EDuration="17.691634189s" totalImagesPullingTime="15.837402386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:20.690172953 +0000 UTC m=+72.897312723" watchObservedRunningTime="2026-09-09 23:49:20.691634189 +0000 UTC m=+72.898773962" Sep 09 23:49:21.698790 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:21.697502 2574 generic.go:356] "Generic (PLEG): container finished" podID="c77e3ae3-38bf-4e62-9a5c-c690eef779d7" containerID="55a451c4ef5b8c68cd9366e2705409ca775ea05b6afa98999c4f901e21498fa2" exitCode=0 Sep 09 23:49:21.698790 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:21.698555 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f6tjk" event={"ID":"c77e3ae3-38bf-4e62-9a5c-c690eef779d7","Type":"ContainerDied","Data":"55a451c4ef5b8c68cd9366e2705409ca775ea05b6afa98999c4f901e21498fa2"} Sep 09 23:49:22.803448 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.803417 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b97bbd854-rf6t6"] Sep 09 23:49:22.823449 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.823419 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b97bbd854-rf6t6"] Sep 09 23:49:22.823627 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.823557 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:22.853941 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.853913 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 23:49:22.938624 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.938209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-oauth-config\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:22.938624 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.938268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-service-ca\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:22.938624 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.938316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-serving-cert\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:22.938624 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.938354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-console-config\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:22.938624 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.938404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-oauth-serving-cert\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:22.938624 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.938439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-trusted-ca-bundle\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:22.938624 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:22.938476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjd9m\" (UniqueName: \"kubernetes.io/projected/b12fc024-71e1-4348-bc6f-0fba911f7551-kube-api-access-mjd9m\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.041658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.038999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-oauth-config\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.041658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.039062 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-service-ca\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.041658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.039096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-serving-cert\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.041658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.039134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-console-config\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.041658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.039181 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-oauth-serving-cert\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.041658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.039214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-trusted-ca-bundle\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.041658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.039255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjd9m\" (UniqueName: \"kubernetes.io/projected/b12fc024-71e1-4348-bc6f-0fba911f7551-kube-api-access-mjd9m\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.041658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.040776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-console-config\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.041658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.041363 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-service-ca\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.043455 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.042806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-oauth-serving-cert\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.043936 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.043887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-trusted-ca-bundle\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.045831 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.045778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-oauth-config\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.047387 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.047346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-serving-cert\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.050420 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.050360 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjd9m\" (UniqueName: \"kubernetes.io/projected/b12fc024-71e1-4348-bc6f-0fba911f7551-kube-api-access-mjd9m\") pod \"console-b97bbd854-rf6t6\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:23.157675 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:23.157537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:24.719941 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:24.719857 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f6tjk" event={"ID":"c77e3ae3-38bf-4e62-9a5c-c690eef779d7","Type":"ContainerStarted","Data":"f9d7dab103c739ddc1705f26514e73a43fb3688a89a00f167962970c6201a090"} Sep 09 23:49:31.210267 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.210240 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b97bbd854-rf6t6"] Sep 09 23:49:31.223536 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:31.223367 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12fc024_71e1_4348_bc6f_0fba911f7551.slice/crio-bbe9275085c58e457c9cc4ccb4f13500a9e6b410b2dc85f0406e376d1d63f740 WatchSource:0}: Error finding container bbe9275085c58e457c9cc4ccb4f13500a9e6b410b2dc85f0406e376d1d63f740: Status 404 returned error can't find the container with id bbe9275085c58e457c9cc4ccb4f13500a9e6b410b2dc85f0406e376d1d63f740 Sep 09 23:49:31.755907 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.755825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f9646598-ttcf5" event={"ID":"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f","Type":"ContainerStarted","Data":"2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e"} Sep 09 23:49:31.757823 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.757797 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gmqgc" event={"ID":"eb88e26c-b785-40be-b781-088af7d86fe2","Type":"ContainerStarted","Data":"0b45d6a6d07e57ecd9f91940d752c3561077aa373e1a8d57ddc1fdb423d74b4d"} Sep 09 23:49:31.757954 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.757845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gmqgc" event={"ID":"eb88e26c-b785-40be-b781-088af7d86fe2","Type":"ContainerStarted","Data":"51dbd0033c5f0ae09e70e054a484a1d432eed9614cb8f358b2233a8c8fd4b3b7"} Sep 09 23:49:31.759377 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.759353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" event={"ID":"4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6","Type":"ContainerStarted","Data":"8a989bf20231d9114d93a4664f3dc38626c626363dca5c0b825297cf8a0b422f"} Sep 09 23:49:31.759475 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.759404 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:31.761029 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.761002 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" event={"ID":"4305d5fc-da8f-4379-805c-b56c2fb77df1","Type":"ContainerStarted","Data":"fd251cd755bfadef35ff70054a16307772062c0e01359a1d007fbc8a8e3b996d"} Sep 09 23:49:31.762505 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.762486 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjngb" event={"ID":"55083d6e-c367-40eb-96ad-b70390fc1039","Type":"ContainerStarted","Data":"7f4a2763e71f5687824155ddfca7fb8691254198f544aa21355fa87e3355025a"} Sep 09 23:49:31.764790 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.764771 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" Sep 09 23:49:31.765026 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.765011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" event={"ID":"7061de57-0809-471e-9535-548e8218cce4","Type":"ContainerStarted","Data":"f39472d174a6384e37bf824d445c18676f1ce85ad4b2e7d62a979651bd365501"} Sep 09 23:49:31.766334 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.766311 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" event={"ID":"a9a255bc-472a-46da-9df5-2a0277aa6369","Type":"ContainerStarted","Data":"bfaf578d2c3f9b9bda2ed91d13775267ff4e569d974c52515fae6fea0d3217ca"} Sep 09 23:49:31.768417 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.768396 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f6tjk" event={"ID":"c77e3ae3-38bf-4e62-9a5c-c690eef779d7","Type":"ContainerStarted","Data":"48b3a29e238bc26ed7d832bef7ea5e79a1d4601ada1b02b2f92339b6b54d7db6"} Sep 09 23:49:31.769749 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.769722 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6mppg" event={"ID":"dd26812f-6704-4784-98a0-a757cc61c378","Type":"ContainerStarted","Data":"4fb1c73e5e3d1a0b90b8e36b2861d01a923a8f67d3ae6900ebb5d299cc12fa63"} Sep 09 23:49:31.769850 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.769792 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:49:31.772263 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.771420 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b97bbd854-rf6t6" event={"ID":"b12fc024-71e1-4348-bc6f-0fba911f7551","Type":"ContainerStarted","Data":"96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd"} Sep 09 23:49:31.772263 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.771462 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b97bbd854-rf6t6" event={"ID":"b12fc024-71e1-4348-bc6f-0fba911f7551","Type":"ContainerStarted","Data":"bbe9275085c58e457c9cc4ccb4f13500a9e6b410b2dc85f0406e376d1d63f740"} Sep 09 23:49:31.773082 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.772418 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f9646598-ttcf5" podStartSLOduration=6.751130022 podStartE2EDuration="17.772402157s" totalImagesPullingTime="11.021272135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:31.771438441 +0000 UTC m=+83.978578212" watchObservedRunningTime="2026-09-09 23:49:31.772402157 +0000 UTC m=+83.979541927" Sep 09 23:49:31.787081 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.787041 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gc7z6" podStartSLOduration=7.973681997 podStartE2EDuration="18.78702939s" totalImagesPullingTime="10.813347393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:31.786198644 +0000 UTC m=+83.993338412" watchObservedRunningTime="2026-09-09 23:49:31.78702939 +0000 UTC m=+83.994169155" Sep 09 23:49:31.800556 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.800522 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gmqgc" podStartSLOduration=72.852363395 podStartE2EDuration="1m23.800510131s" totalImagesPullingTime="10.948146736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:31.799516451 +0000 UTC m=+84.006656222" watchObservedRunningTime="2026-09-09 23:49:31.800510131 +0000 UTC m=+84.007649910" Sep 09 23:49:31.817404 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.817370 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f6tjk" podStartSLOduration=17.475699527 podStartE2EDuration="18.817357608s" totalImagesPullingTime="1.222937903s" totalInitContainerRuntime="118.720178ms" isStatefulPod=true podCreationTimestamp="2026-09-09 23:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:31.816730057 +0000 UTC m=+84.023869826" watchObservedRunningTime="2026-09-09 23:49:31.817357608 +0000 UTC m=+84.024497427" Sep 09 23:49:31.834549 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.834515 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" podStartSLOduration=3.907234539 podStartE2EDuration="14.834503051s" totalImagesPullingTime="10.927268512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:31.833636439 +0000 UTC m=+84.040776220" watchObservedRunningTime="2026-09-09 23:49:31.834503051 +0000 UTC m=+84.041642831" Sep 09 23:49:31.854011 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.853975 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6mppg" podStartSLOduration=72.804609988 podStartE2EDuration="1m23.853964343s" totalImagesPullingTime="11.049354355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:31.853165471 +0000 UTC m=+84.060305240" watchObservedRunningTime="2026-09-09 23:49:31.853964343 +0000 UTC m=+84.061104118" Sep 09 23:49:31.868744 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.868710 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-kxbpp" podStartSLOduration=2.901496 podStartE2EDuration="13.868698825s" totalImagesPullingTime="10.967202825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:31.867673996 +0000 UTC m=+84.074813766" watchObservedRunningTime="2026-09-09 23:49:31.868698825 +0000 UTC m=+84.075838593" Sep 09 23:49:31.882000 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.881953 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bjngb" podStartSLOduration=72.68474788 podStartE2EDuration="1m23.881936753s" totalImagesPullingTime="11.197188873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:31.881154789 +0000 UTC m=+84.088294562" watchObservedRunningTime="2026-09-09 23:49:31.881936753 +0000 UTC m=+84.089076535" Sep 09 23:49:31.909268 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:31.909216 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b97bbd854-rf6t6" podStartSLOduration=9.909199159 podStartE2EDuration="9.909199159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:31.907572213 +0000 UTC m=+84.114711982" watchObservedRunningTime="2026-09-09 23:49:31.909199159 +0000 UTC m=+84.116338930" Sep 09 23:49:33.157878 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.157779 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:33.157878 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.157870 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:33.162803 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.162782 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:33.781248 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.781193 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" event={"ID":"4305d5fc-da8f-4379-805c-b56c2fb77df1","Type":"ContainerStarted","Data":"18f014e808c40e8d2c3a4b6be2b786410fdaa55048eb9bc50f48dc98b14fa934"} Sep 09 23:49:33.781477 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.781296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" event={"ID":"4305d5fc-da8f-4379-805c-b56c2fb77df1","Type":"ContainerStarted","Data":"c6930625f5a8bac497dd3b02906d691563adb3aba3a120e6fe6956c9e3ad4819"} Sep 09 23:49:33.785382 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.785359 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:49:33.801911 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.801870 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6b5787994-8f97s" podStartSLOduration=3.287093921 podStartE2EDuration="15.801853967s" totalImagesPullingTime="12.514760046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:33.80010587 +0000 UTC m=+86.007245638" watchObservedRunningTime="2026-09-09 23:49:33.801853967 +0000 UTC m=+86.008993736" Sep 09 23:49:33.840538 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.840514 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f9646598-ttcf5"] Sep 09 23:49:33.939239 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.939209 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86cdcb799c-dfxhn"] Sep 09 23:49:33.974733 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.974709 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cdcb799c-dfxhn"] Sep 09 23:49:33.974854 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:33.974800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.037339 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.037317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-oauth-config\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.037442 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.037356 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-oauth-serving-cert\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.037442 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.037386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-kube-api-access-hjgmr\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.037442 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.037406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-config\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.037442 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.037439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-service-ca\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.037571 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.037472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-trusted-ca-bundle\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.037571 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.037492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-serving-cert\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.137958 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.137928 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-trusted-ca-bundle\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.137958 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.137959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-serving-cert\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.138129 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.137989 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-oauth-config\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.138129 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.138021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-oauth-serving-cert\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.138129 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.138049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-kube-api-access-hjgmr\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.138129 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.138074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-config\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.138129 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.138106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-service-ca\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.138867 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.138845 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-oauth-serving-cert\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.139030 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.138883 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-service-ca\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.139030 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.138904 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-config\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.139142 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.138944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-trusted-ca-bundle\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.140498 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.140479 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-oauth-config\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.140597 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.140577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-serving-cert\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.145683 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.145662 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-kube-api-access-hjgmr\") pod \"console-86cdcb799c-dfxhn\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.283474 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.283445 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:34.579731 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.579708 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cdcb799c-dfxhn"] Sep 09 23:49:34.582009 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:49:34.581986 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ca0aed_9d4a_4211_bfde_3b41d042d6d3.slice/crio-3a398b0899f86302a4ffe7baf4abc1b3b3cd121e00b0e0fb91a2001516d61016 WatchSource:0}: Error finding container 3a398b0899f86302a4ffe7baf4abc1b3b3cd121e00b0e0fb91a2001516d61016: Status 404 returned error can't find the container with id 3a398b0899f86302a4ffe7baf4abc1b3b3cd121e00b0e0fb91a2001516d61016 Sep 09 23:49:34.602755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.602737 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:34.787217 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.787178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cdcb799c-dfxhn" event={"ID":"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3","Type":"ContainerStarted","Data":"b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758"} Sep 09 23:49:34.787398 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.787231 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cdcb799c-dfxhn" event={"ID":"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3","Type":"ContainerStarted","Data":"3a398b0899f86302a4ffe7baf4abc1b3b3cd121e00b0e0fb91a2001516d61016"} Sep 09 23:49:34.807906 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:34.804362 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86cdcb799c-dfxhn" podStartSLOduration=1.80434294 podStartE2EDuration="1.80434294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:34.803051748 +0000 UTC m=+87.010191508" watchObservedRunningTime="2026-09-09 23:49:34.80434294 +0000 UTC m=+87.011482710" Sep 09 23:49:38.034735 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:38.034701 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:38.035104 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:38.034759 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:44.284465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:44.284433 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:44.284953 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:44.284512 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:44.289181 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:44.289163 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:44.828536 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:44.828508 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:49:44.869764 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:44.869732 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b97bbd854-rf6t6"] Sep 09 23:49:58.040475 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:58.040441 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:58.044318 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:58.044295 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-587c665f98-fhqhx" Sep 09 23:49:58.859393 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:58.859336 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f9646598-ttcf5" podUID="950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" containerName="console" containerID="cri-o://2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e" gracePeriod=15 Sep 09 23:49:59.121482 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.121407 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:59.222124 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.222089 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-serving-cert\") pod \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " Sep 09 23:49:59.222124 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.222131 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-config\") pod \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " Sep 09 23:49:59.222393 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.222154 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-service-ca\") pod \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " Sep 09 23:49:59.222393 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.222208 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2brg\" (UniqueName: \"kubernetes.io/projected/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-kube-api-access-h2brg\") pod \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " Sep 09 23:49:59.222393 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.222240 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-oauth-config\") pod \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " Sep 09 23:49:59.222393 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.222260 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-oauth-serving-cert\") pod \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\" (UID: \"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f\") " Sep 09 23:49:59.222616 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.222581 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-config" (OuterVolumeSpecName: "console-config") pod "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" (UID: "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:49:59.222738 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.222596 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-service-ca" (OuterVolumeSpecName: "service-ca") pod "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" (UID: "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:49:59.222738 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.222713 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" (UID: "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:49:59.224638 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.224605 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" (UID: "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:49:59.224747 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.224680 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-kube-api-access-h2brg" (OuterVolumeSpecName: "kube-api-access-h2brg") pod "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" (UID: "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f"). InnerVolumeSpecName "kube-api-access-h2brg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:49:59.224747 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.224687 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" (UID: "950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:49:59.323606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.323577 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-oauth-config\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:49:59.323606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.323606 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-oauth-serving-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:49:59.323767 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.323616 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-serving-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:49:59.323767 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.323625 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-console-config\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:49:59.323767 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.323633 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-service-ca\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:49:59.323767 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.323641 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2brg\" (UniqueName: \"kubernetes.io/projected/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f-kube-api-access-h2brg\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:49:59.881297 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.881256 2574 generic.go:356] "Generic (PLEG): container finished" podID="950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" containerID="2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e" exitCode=0 Sep 09 23:49:59.881466 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.881324 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f9646598-ttcf5" Sep 09 23:49:59.881466 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.881341 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f9646598-ttcf5" event={"ID":"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f","Type":"ContainerDied","Data":"2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e"} Sep 09 23:49:59.881466 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.881395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f9646598-ttcf5" event={"ID":"950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f","Type":"ContainerDied","Data":"40bd6aeaea6609f32d0c08c1c523e3b774d573cfc46902ed4a7852b691207828"} Sep 09 23:49:59.881466 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.881437 2574 scope.go:117] "RemoveContainer" containerID="2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e" Sep 09 23:49:59.889369 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.889349 2574 scope.go:117] "RemoveContainer" containerID="2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e" Sep 09 23:49:59.889659 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:49:59.889640 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e\": container with ID starting with 2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e not found: ID does not exist" containerID="2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e" Sep 09 23:49:59.889736 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.889665 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e"} err="failed to get container status \"2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e\": rpc error: code = NotFound desc = could not find container \"2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e\": container with ID starting with 2523cbb039994e86890baaa87bc7e5bec32607453559fcdaedacf20af78d6e3e not found: ID does not exist" Sep 09 23:49:59.900537 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.900506 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f9646598-ttcf5"] Sep 09 23:49:59.905094 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:49:59.905073 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f9646598-ttcf5"] Sep 09 23:50:00.238098 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:00.238012 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" path="/var/lib/kubelet/pods/950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f/volumes" Sep 09 23:50:02.779430 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:02.779398 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6mppg" Sep 09 23:50:09.889474 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:09.889402 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b97bbd854-rf6t6" podUID="b12fc024-71e1-4348-bc6f-0fba911f7551" containerName="console" containerID="cri-o://96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd" gracePeriod=15 Sep 09 23:50:10.165902 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.165845 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:50:10.204404 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.204380 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-oauth-config\") pod \"b12fc024-71e1-4348-bc6f-0fba911f7551\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " Sep 09 23:50:10.204404 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.204415 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-console-config\") pod \"b12fc024-71e1-4348-bc6f-0fba911f7551\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " Sep 09 23:50:10.204591 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.204441 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-serving-cert\") pod \"b12fc024-71e1-4348-bc6f-0fba911f7551\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " Sep 09 23:50:10.204591 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.204461 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-service-ca\") pod \"b12fc024-71e1-4348-bc6f-0fba911f7551\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " Sep 09 23:50:10.204591 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.204486 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjd9m\" (UniqueName: \"kubernetes.io/projected/b12fc024-71e1-4348-bc6f-0fba911f7551-kube-api-access-mjd9m\") pod \"b12fc024-71e1-4348-bc6f-0fba911f7551\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " Sep 09 23:50:10.204591 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.204513 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-oauth-serving-cert\") pod \"b12fc024-71e1-4348-bc6f-0fba911f7551\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " Sep 09 23:50:10.204591 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.204537 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-trusted-ca-bundle\") pod \"b12fc024-71e1-4348-bc6f-0fba911f7551\" (UID: \"b12fc024-71e1-4348-bc6f-0fba911f7551\") " Sep 09 23:50:10.205025 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.204995 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b12fc024-71e1-4348-bc6f-0fba911f7551" (UID: "b12fc024-71e1-4348-bc6f-0fba911f7551"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:50:10.205138 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.205016 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-service-ca" (OuterVolumeSpecName: "service-ca") pod "b12fc024-71e1-4348-bc6f-0fba911f7551" (UID: "b12fc024-71e1-4348-bc6f-0fba911f7551"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:50:10.205138 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.205120 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-console-config" (OuterVolumeSpecName: "console-config") pod "b12fc024-71e1-4348-bc6f-0fba911f7551" (UID: "b12fc024-71e1-4348-bc6f-0fba911f7551"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:50:10.205266 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.205161 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b12fc024-71e1-4348-bc6f-0fba911f7551" (UID: "b12fc024-71e1-4348-bc6f-0fba911f7551"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:50:10.206578 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.206550 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b12fc024-71e1-4348-bc6f-0fba911f7551" (UID: "b12fc024-71e1-4348-bc6f-0fba911f7551"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:50:10.206704 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.206686 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12fc024-71e1-4348-bc6f-0fba911f7551-kube-api-access-mjd9m" (OuterVolumeSpecName: "kube-api-access-mjd9m") pod "b12fc024-71e1-4348-bc6f-0fba911f7551" (UID: "b12fc024-71e1-4348-bc6f-0fba911f7551"). InnerVolumeSpecName "kube-api-access-mjd9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:50:10.206749 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.206691 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b12fc024-71e1-4348-bc6f-0fba911f7551" (UID: "b12fc024-71e1-4348-bc6f-0fba911f7551"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:50:10.306044 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.306009 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-oauth-config\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:50:10.306044 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.306038 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-console-config\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:50:10.306044 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.306048 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b12fc024-71e1-4348-bc6f-0fba911f7551-console-serving-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:50:10.306265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.306056 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-service-ca\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:50:10.306265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.306064 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjd9m\" (UniqueName: \"kubernetes.io/projected/b12fc024-71e1-4348-bc6f-0fba911f7551-kube-api-access-mjd9m\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:50:10.306265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.306073 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-oauth-serving-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:50:10.306265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.306081 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12fc024-71e1-4348-bc6f-0fba911f7551-trusted-ca-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:50:10.921944 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.921908 2574 generic.go:356] "Generic (PLEG): container finished" podID="b12fc024-71e1-4348-bc6f-0fba911f7551" containerID="96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd" exitCode=0 Sep 09 23:50:10.922420 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.921955 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b97bbd854-rf6t6" Sep 09 23:50:10.922420 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.921975 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b97bbd854-rf6t6" event={"ID":"b12fc024-71e1-4348-bc6f-0fba911f7551","Type":"ContainerDied","Data":"96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd"} Sep 09 23:50:10.922420 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.922025 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b97bbd854-rf6t6" event={"ID":"b12fc024-71e1-4348-bc6f-0fba911f7551","Type":"ContainerDied","Data":"bbe9275085c58e457c9cc4ccb4f13500a9e6b410b2dc85f0406e376d1d63f740"} Sep 09 23:50:10.922420 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.922054 2574 scope.go:117] "RemoveContainer" containerID="96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd" Sep 09 23:50:10.930018 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.930001 2574 scope.go:117] "RemoveContainer" containerID="96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd" Sep 09 23:50:10.930292 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:50:10.930261 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd\": container with ID starting with 96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd not found: ID does not exist" containerID="96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd" Sep 09 23:50:10.930405 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.930386 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd"} err="failed to get container status \"96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd\": rpc error: code = NotFound desc = could not find container \"96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd\": container with ID starting with 96b3277a5fc745eafc3b21857f518a494cc393bd64ab270e1cfaa8299b51f3cd not found: ID does not exist" Sep 09 23:50:10.939626 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.937583 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b97bbd854-rf6t6"] Sep 09 23:50:10.941178 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:10.941156 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b97bbd854-rf6t6"] Sep 09 23:50:12.237818 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:12.237781 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12fc024-71e1-4348-bc6f-0fba911f7551" path="/var/lib/kubelet/pods/b12fc024-71e1-4348-bc6f-0fba911f7551/volumes" Sep 09 23:50:37.127031 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.126996 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-575fd5647d-9t52x"] Sep 09 23:50:37.127487 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.127330 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12fc024-71e1-4348-bc6f-0fba911f7551" containerName="console" Sep 09 23:50:37.127487 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.127347 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12fc024-71e1-4348-bc6f-0fba911f7551" containerName="console" Sep 09 23:50:37.127487 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.127363 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" containerName="console" Sep 09 23:50:37.127487 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.127368 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" containerName="console" Sep 09 23:50:37.127487 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.127423 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12fc024-71e1-4348-bc6f-0fba911f7551" containerName="console" Sep 09 23:50:37.127487 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.127437 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="950d9a0c-3687-4a5a-8d76-6b2b0d4dd19f" containerName="console" Sep 09 23:50:37.129339 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.129322 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.140458 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.140435 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575fd5647d-9t52x"] Sep 09 23:50:37.196850 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.196819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-oauth-config\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.196987 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.196857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-oauth-serving-cert\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.196987 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.196884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-trusted-ca-bundle\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.196987 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.196960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml9vs\" (UniqueName: \"kubernetes.io/projected/f9c58547-94e1-4618-9b52-d48b9d267427-kube-api-access-ml9vs\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.197089 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.196995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-service-ca\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.197089 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.197022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-serving-cert\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.197089 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.197041 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-console-config\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.298360 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.298328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-service-ca\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.298522 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.298371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-serving-cert\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.298522 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.298387 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-console-config\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.298614 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.298518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-oauth-config\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.298614 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.298572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-oauth-serving-cert\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.298716 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.298622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-trusted-ca-bundle\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.298716 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.298654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml9vs\" (UniqueName: \"kubernetes.io/projected/f9c58547-94e1-4618-9b52-d48b9d267427-kube-api-access-ml9vs\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.299157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.299130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-service-ca\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.299365 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.299205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-console-config\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.299365 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.299358 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-oauth-serving-cert\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.299534 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.299459 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-trusted-ca-bundle\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.300889 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.300865 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-serving-cert\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.300981 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.300949 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-oauth-config\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.305626 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.305605 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml9vs\" (UniqueName: \"kubernetes.io/projected/f9c58547-94e1-4618-9b52-d48b9d267427-kube-api-access-ml9vs\") pod \"console-575fd5647d-9t52x\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.440827 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.440750 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:37.709415 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:37.709389 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575fd5647d-9t52x"] Sep 09 23:50:37.711955 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:50:37.711928 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c58547_94e1_4618_9b52_d48b9d267427.slice/crio-17bd5e20d5140b499655aef038b16aed3565d4f0e2084bb13d7e61a19fcc0bf5 WatchSource:0}: Error finding container 17bd5e20d5140b499655aef038b16aed3565d4f0e2084bb13d7e61a19fcc0bf5: Status 404 returned error can't find the container with id 17bd5e20d5140b499655aef038b16aed3565d4f0e2084bb13d7e61a19fcc0bf5 Sep 09 23:50:38.013751 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:38.013666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575fd5647d-9t52x" event={"ID":"f9c58547-94e1-4618-9b52-d48b9d267427","Type":"ContainerStarted","Data":"0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e"} Sep 09 23:50:38.013751 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:38.013729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575fd5647d-9t52x" event={"ID":"f9c58547-94e1-4618-9b52-d48b9d267427","Type":"ContainerStarted","Data":"17bd5e20d5140b499655aef038b16aed3565d4f0e2084bb13d7e61a19fcc0bf5"} Sep 09 23:50:38.039166 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:38.039117 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-575fd5647d-9t52x" podStartSLOduration=1.039101766 podStartE2EDuration="1.039101766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:50:38.038595116 +0000 UTC m=+150.245734900" watchObservedRunningTime="2026-09-09 23:50:38.039101766 +0000 UTC m=+150.246241547" Sep 09 23:50:47.441545 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:47.441512 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:47.441979 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:47.441563 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:47.446323 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:47.446299 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:48.051970 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:48.051940 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:50:48.087844 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:50:48.087814 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cdcb799c-dfxhn"] Sep 09 23:51:13.107455 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.107394 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86cdcb799c-dfxhn" podUID="b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" containerName="console" containerID="cri-o://b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758" gracePeriod=15 Sep 09 23:51:13.342321 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.342299 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:51:13.475822 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.475739 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-service-ca\") pod \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " Sep 09 23:51:13.475822 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.475787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-oauth-serving-cert\") pod \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " Sep 09 23:51:13.475822 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.475815 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-oauth-config\") pod \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " Sep 09 23:51:13.476071 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.475851 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-kube-api-access-hjgmr\") pod \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " Sep 09 23:51:13.476071 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.475916 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-serving-cert\") pod \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " Sep 09 23:51:13.476071 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.475939 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-trusted-ca-bundle\") pod \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " Sep 09 23:51:13.476071 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.475963 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-config\") pod \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\" (UID: \"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3\") " Sep 09 23:51:13.476265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.476168 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-service-ca" (OuterVolumeSpecName: "service-ca") pod "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" (UID: "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:51:13.476348 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.476312 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" (UID: "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:51:13.476406 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.476343 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" (UID: "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:51:13.476505 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.476461 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-config" (OuterVolumeSpecName: "console-config") pod "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" (UID: "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:51:13.478029 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.478001 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" (UID: "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:51:13.478116 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.478037 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-kube-api-access-hjgmr" (OuterVolumeSpecName: "kube-api-access-hjgmr") pod "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" (UID: "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3"). InnerVolumeSpecName "kube-api-access-hjgmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:51:13.478116 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.478052 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" (UID: "b5ca0aed-9d4a-4211-bfde-3b41d042d6d3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:51:13.577272 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.577248 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-service-ca\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:13.577272 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.577269 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-oauth-serving-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:13.577407 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.577289 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-oauth-config\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:13.577407 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.577300 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-kube-api-access-hjgmr\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:13.577407 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.577309 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-serving-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:13.577407 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.577317 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-trusted-ca-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:13.577407 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:13.577325 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3-console-config\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:14.142604 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.142566 2574 generic.go:356] "Generic (PLEG): container finished" podID="b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" containerID="b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758" exitCode=0 Sep 09 23:51:14.143072 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.142616 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cdcb799c-dfxhn" Sep 09 23:51:14.143072 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.142655 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cdcb799c-dfxhn" event={"ID":"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3","Type":"ContainerDied","Data":"b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758"} Sep 09 23:51:14.143072 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.142724 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cdcb799c-dfxhn" event={"ID":"b5ca0aed-9d4a-4211-bfde-3b41d042d6d3","Type":"ContainerDied","Data":"3a398b0899f86302a4ffe7baf4abc1b3b3cd121e00b0e0fb91a2001516d61016"} Sep 09 23:51:14.143072 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.142768 2574 scope.go:117] "RemoveContainer" containerID="b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758" Sep 09 23:51:14.150968 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.150688 2574 scope.go:117] "RemoveContainer" containerID="b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758" Sep 09 23:51:14.151038 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:51:14.151011 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758\": container with ID starting with b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758 not found: ID does not exist" containerID="b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758" Sep 09 23:51:14.151089 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.151045 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758"} err="failed to get container status \"b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758\": rpc error: code = NotFound desc = could not find container \"b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758\": container with ID starting with b3effa96c4a54686a9f9773f61bdfad3c2d050f3914d30ea5b0dfbe867430758 not found: ID does not exist" Sep 09 23:51:14.162403 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.162378 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cdcb799c-dfxhn"] Sep 09 23:51:14.164003 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.163983 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86cdcb799c-dfxhn"] Sep 09 23:51:14.238291 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:14.238256 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" path="/var/lib/kubelet/pods/b5ca0aed-9d4a-4211-bfde-3b41d042d6d3/volumes" Sep 09 23:51:26.690350 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.690319 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn"] Sep 09 23:51:26.690808 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.690584 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" containerName="console" Sep 09 23:51:26.690808 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.690599 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" containerName="console" Sep 09 23:51:26.690808 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.690653 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ca0aed-9d4a-4211-bfde-3b41d042d6d3" containerName="console" Sep 09 23:51:26.693189 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.693174 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:26.695413 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.695357 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-67ntj\"" Sep 09 23:51:26.695614 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.695598 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 23:51:26.696170 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.696151 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 23:51:26.701237 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.701215 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn"] Sep 09 23:51:26.864996 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.864964 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:26.865142 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.865004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6p2t\" (UniqueName: \"kubernetes.io/projected/cfa85cf4-6e08-446f-b0a3-603137a01ff1-kube-api-access-x6p2t\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:26.865142 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.865045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:26.966335 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.966252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:26.966335 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.966307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6p2t\" (UniqueName: \"kubernetes.io/projected/cfa85cf4-6e08-446f-b0a3-603137a01ff1-kube-api-access-x6p2t\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:26.966482 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.966341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:26.966685 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.966664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:26.966723 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.966669 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:26.973842 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:26.973814 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6p2t\" (UniqueName: \"kubernetes.io/projected/cfa85cf4-6e08-446f-b0a3-603137a01ff1-kube-api-access-x6p2t\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:27.002704 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:27.002675 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:27.266105 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:27.266030 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn"] Sep 09 23:51:27.269088 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:51:27.269060 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa85cf4_6e08_446f_b0a3_603137a01ff1.slice/crio-d026dd33b3d92201a020c8cb0211021f4da3ed67b9bf97ea3723024aa0754dac WatchSource:0}: Error finding container d026dd33b3d92201a020c8cb0211021f4da3ed67b9bf97ea3723024aa0754dac: Status 404 returned error can't find the container with id d026dd33b3d92201a020c8cb0211021f4da3ed67b9bf97ea3723024aa0754dac Sep 09 23:51:28.193236 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:28.193200 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" event={"ID":"cfa85cf4-6e08-446f-b0a3-603137a01ff1","Type":"ContainerStarted","Data":"d026dd33b3d92201a020c8cb0211021f4da3ed67b9bf97ea3723024aa0754dac"} Sep 09 23:51:35.219884 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:35.219852 2574 generic.go:356] "Generic (PLEG): container finished" podID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerID="09657b10ca866e9bd78b47834d12bfbc4420427f69fd298ddda8c9bb907b98e6" exitCode=0 Sep 09 23:51:35.220267 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:35.219918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" event={"ID":"cfa85cf4-6e08-446f-b0a3-603137a01ff1","Type":"ContainerDied","Data":"09657b10ca866e9bd78b47834d12bfbc4420427f69fd298ddda8c9bb907b98e6"} Sep 09 23:51:37.230395 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:37.230365 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" event={"ID":"cfa85cf4-6e08-446f-b0a3-603137a01ff1","Type":"ContainerStarted","Data":"18f53bf23145ad95fbce09441abaab3f4c166cd40aecbcc268c0227997fdfd23"} Sep 09 23:51:38.240821 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:38.240782 2574 generic.go:356] "Generic (PLEG): container finished" podID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerID="18f53bf23145ad95fbce09441abaab3f4c166cd40aecbcc268c0227997fdfd23" exitCode=0 Sep 09 23:51:38.242425 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:38.242394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" event={"ID":"cfa85cf4-6e08-446f-b0a3-603137a01ff1","Type":"ContainerDied","Data":"18f53bf23145ad95fbce09441abaab3f4c166cd40aecbcc268c0227997fdfd23"} Sep 09 23:51:45.272731 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:45.272645 2574 generic.go:356] "Generic (PLEG): container finished" podID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerID="8b0fa0250a227f48c37b5b9255e1d69acf0a3eecf63331615f001e6f1a708548" exitCode=0 Sep 09 23:51:45.273082 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:45.272730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" event={"ID":"cfa85cf4-6e08-446f-b0a3-603137a01ff1","Type":"ContainerDied","Data":"8b0fa0250a227f48c37b5b9255e1d69acf0a3eecf63331615f001e6f1a708548"} Sep 09 23:51:46.392701 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.392677 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:46.404192 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.404168 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-util\") pod \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " Sep 09 23:51:46.404301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.404209 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6p2t\" (UniqueName: \"kubernetes.io/projected/cfa85cf4-6e08-446f-b0a3-603137a01ff1-kube-api-access-x6p2t\") pod \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " Sep 09 23:51:46.404301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.404267 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-bundle\") pod \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\" (UID: \"cfa85cf4-6e08-446f-b0a3-603137a01ff1\") " Sep 09 23:51:46.404947 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.404921 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-bundle" (OuterVolumeSpecName: "bundle") pod "cfa85cf4-6e08-446f-b0a3-603137a01ff1" (UID: "cfa85cf4-6e08-446f-b0a3-603137a01ff1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:51:46.406363 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.406335 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa85cf4-6e08-446f-b0a3-603137a01ff1-kube-api-access-x6p2t" (OuterVolumeSpecName: "kube-api-access-x6p2t") pod "cfa85cf4-6e08-446f-b0a3-603137a01ff1" (UID: "cfa85cf4-6e08-446f-b0a3-603137a01ff1"). InnerVolumeSpecName "kube-api-access-x6p2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:51:46.408909 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.408885 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-util" (OuterVolumeSpecName: "util") pod "cfa85cf4-6e08-446f-b0a3-603137a01ff1" (UID: "cfa85cf4-6e08-446f-b0a3-603137a01ff1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:51:46.505045 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.505016 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6p2t\" (UniqueName: \"kubernetes.io/projected/cfa85cf4-6e08-446f-b0a3-603137a01ff1-kube-api-access-x6p2t\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:46.505045 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.505039 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:46.505045 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:46.505049 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa85cf4-6e08-446f-b0a3-603137a01ff1-util\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:51:47.280440 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:47.280407 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" Sep 09 23:51:47.280601 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:47.280404 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407x7wn" event={"ID":"cfa85cf4-6e08-446f-b0a3-603137a01ff1","Type":"ContainerDied","Data":"d026dd33b3d92201a020c8cb0211021f4da3ed67b9bf97ea3723024aa0754dac"} Sep 09 23:51:47.280601 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:47.280539 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d026dd33b3d92201a020c8cb0211021f4da3ed67b9bf97ea3723024aa0754dac" Sep 09 23:51:54.667064 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.667032 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c"] Sep 09 23:51:54.667469 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.667278 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerName="pull" Sep 09 23:51:54.667469 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.667304 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerName="pull" Sep 09 23:51:54.667469 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.667313 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerName="util" Sep 09 23:51:54.667469 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.667319 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerName="util" Sep 09 23:51:54.667469 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.667329 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerName="extract" Sep 09 23:51:54.667469 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.667335 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerName="extract" Sep 09 23:51:54.667469 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.667379 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa85cf4-6e08-446f-b0a3-603137a01ff1" containerName="extract" Sep 09 23:51:54.670667 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.670649 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" Sep 09 23:51:54.673632 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.673609 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:51:54.673744 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.673638 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 09 23:51:54.673744 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.673609 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dl49k\"" Sep 09 23:51:54.682449 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.682425 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c"] Sep 09 23:51:54.760690 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.760653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wc5l\" (UniqueName: \"kubernetes.io/projected/fc422bbd-c878-46d1-825f-9e4b35993e49-kube-api-access-6wc5l\") pod \"cert-manager-operator-controller-manager-55c444bf86-z962c\" (UID: \"fc422bbd-c878-46d1-825f-9e4b35993e49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" Sep 09 23:51:54.760690 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.760693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc422bbd-c878-46d1-825f-9e4b35993e49-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-z962c\" (UID: \"fc422bbd-c878-46d1-825f-9e4b35993e49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" Sep 09 23:51:54.861117 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.861079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wc5l\" (UniqueName: \"kubernetes.io/projected/fc422bbd-c878-46d1-825f-9e4b35993e49-kube-api-access-6wc5l\") pod \"cert-manager-operator-controller-manager-55c444bf86-z962c\" (UID: \"fc422bbd-c878-46d1-825f-9e4b35993e49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" Sep 09 23:51:54.861332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.861127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc422bbd-c878-46d1-825f-9e4b35993e49-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-z962c\" (UID: \"fc422bbd-c878-46d1-825f-9e4b35993e49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" Sep 09 23:51:54.861518 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.861498 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc422bbd-c878-46d1-825f-9e4b35993e49-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-z962c\" (UID: \"fc422bbd-c878-46d1-825f-9e4b35993e49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" Sep 09 23:51:54.869658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.869638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wc5l\" (UniqueName: \"kubernetes.io/projected/fc422bbd-c878-46d1-825f-9e4b35993e49-kube-api-access-6wc5l\") pod \"cert-manager-operator-controller-manager-55c444bf86-z962c\" (UID: \"fc422bbd-c878-46d1-825f-9e4b35993e49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" Sep 09 23:51:54.979714 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:54.979631 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" Sep 09 23:51:55.255516 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:55.255435 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c"] Sep 09 23:51:55.258652 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:51:55.258625 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc422bbd_c878_46d1_825f_9e4b35993e49.slice/crio-1712710f6689687f4e553fc793d1f80820dbd8bdb3c9234938e41d6c0f42939f WatchSource:0}: Error finding container 1712710f6689687f4e553fc793d1f80820dbd8bdb3c9234938e41d6c0f42939f: Status 404 returned error can't find the container with id 1712710f6689687f4e553fc793d1f80820dbd8bdb3c9234938e41d6c0f42939f Sep 09 23:51:55.310116 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:55.310088 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" event={"ID":"fc422bbd-c878-46d1-825f-9e4b35993e49","Type":"ContainerStarted","Data":"1712710f6689687f4e553fc793d1f80820dbd8bdb3c9234938e41d6c0f42939f"} Sep 09 23:51:57.320543 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:57.320502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" event={"ID":"fc422bbd-c878-46d1-825f-9e4b35993e49","Type":"ContainerStarted","Data":"7a6bcc6fca3e9eab8a4550523222b6fbc55301f8e8b1d582b4832e5f1da95f83"} Sep 09 23:51:57.341628 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:57.341578 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-z962c" podStartSLOduration=1.415102891 podStartE2EDuration="3.34156177s" totalImagesPullingTime="1.926458879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:51:57.339273483 +0000 UTC m=+229.546413278" watchObservedRunningTime="2026-09-09 23:51:57.34156177 +0000 UTC m=+229.548701539" Sep 09 23:51:58.404048 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.404019 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zgkwz"] Sep 09 23:51:58.405973 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.405959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:51:58.408382 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.408361 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-x9jf7\"" Sep 09 23:51:58.408382 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.408373 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 09 23:51:58.409101 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.409087 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 09 23:51:58.416065 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.416044 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zgkwz"] Sep 09 23:51:58.486156 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.486123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2d4\" (UniqueName: \"kubernetes.io/projected/9524c44e-5e0f-45c7-bd16-7e086bee0370-kube-api-access-6c2d4\") pod \"cert-manager-webhook-6486ffd997-zgkwz\" (UID: \"9524c44e-5e0f-45c7-bd16-7e086bee0370\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:51:58.486156 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.486161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9524c44e-5e0f-45c7-bd16-7e086bee0370-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zgkwz\" (UID: \"9524c44e-5e0f-45c7-bd16-7e086bee0370\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:51:58.586791 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.586757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2d4\" (UniqueName: \"kubernetes.io/projected/9524c44e-5e0f-45c7-bd16-7e086bee0370-kube-api-access-6c2d4\") pod \"cert-manager-webhook-6486ffd997-zgkwz\" (UID: \"9524c44e-5e0f-45c7-bd16-7e086bee0370\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:51:58.586919 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.586800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9524c44e-5e0f-45c7-bd16-7e086bee0370-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zgkwz\" (UID: \"9524c44e-5e0f-45c7-bd16-7e086bee0370\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:51:58.595602 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.595572 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9524c44e-5e0f-45c7-bd16-7e086bee0370-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zgkwz\" (UID: \"9524c44e-5e0f-45c7-bd16-7e086bee0370\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:51:58.595706 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.595617 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2d4\" (UniqueName: \"kubernetes.io/projected/9524c44e-5e0f-45c7-bd16-7e086bee0370-kube-api-access-6c2d4\") pod \"cert-manager-webhook-6486ffd997-zgkwz\" (UID: \"9524c44e-5e0f-45c7-bd16-7e086bee0370\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:51:58.640033 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.640004 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5"] Sep 09 23:51:58.642257 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.642242 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.644528 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.644507 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 23:51:58.644646 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.644590 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 23:51:58.644948 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.644929 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-67ntj\"" Sep 09 23:51:58.651573 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.651555 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5"] Sep 09 23:51:58.687566 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.687501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.687566 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.687546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.687713 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.687580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55t86\" (UniqueName: \"kubernetes.io/projected/8d2c2841-234c-4d20-b6d9-a9259966e089-kube-api-access-55t86\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.722645 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.722613 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:51:58.788075 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.788046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.788209 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.788096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.788209 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.788133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55t86\" (UniqueName: \"kubernetes.io/projected/8d2c2841-234c-4d20-b6d9-a9259966e089-kube-api-access-55t86\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.788554 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.788525 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.788554 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.788550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.796065 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.796038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55t86\" (UniqueName: \"kubernetes.io/projected/8d2c2841-234c-4d20-b6d9-a9259966e089-kube-api-access-55t86\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.951927 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.951848 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:51:58.986837 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:58.986775 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zgkwz"] Sep 09 23:51:58.989408 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:51:58.989378 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9524c44e_5e0f_45c7_bd16_7e086bee0370.slice/crio-857501886e588464942ef1105ed8a0d9fee8dd42269b8be38146a60a179cc296 WatchSource:0}: Error finding container 857501886e588464942ef1105ed8a0d9fee8dd42269b8be38146a60a179cc296: Status 404 returned error can't find the container with id 857501886e588464942ef1105ed8a0d9fee8dd42269b8be38146a60a179cc296 Sep 09 23:51:59.221357 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:59.221135 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5"] Sep 09 23:51:59.223679 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:51:59.223649 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2c2841_234c_4d20_b6d9_a9259966e089.slice/crio-b4e8aaff3cb4701a2f7b8cf8ae0520d26ee3ea5a04548e3f720725ee4e2c209d WatchSource:0}: Error finding container b4e8aaff3cb4701a2f7b8cf8ae0520d26ee3ea5a04548e3f720725ee4e2c209d: Status 404 returned error can't find the container with id b4e8aaff3cb4701a2f7b8cf8ae0520d26ee3ea5a04548e3f720725ee4e2c209d Sep 09 23:51:59.330650 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:59.330623 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerID="46b2d3d1e38e7183aea7e85d0db20beb33c7aa05cd10303fb2b100721232dcab" exitCode=0 Sep 09 23:51:59.330754 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:59.330701 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" event={"ID":"8d2c2841-234c-4d20-b6d9-a9259966e089","Type":"ContainerDied","Data":"46b2d3d1e38e7183aea7e85d0db20beb33c7aa05cd10303fb2b100721232dcab"} Sep 09 23:51:59.330823 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:59.330752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" event={"ID":"8d2c2841-234c-4d20-b6d9-a9259966e089","Type":"ContainerStarted","Data":"b4e8aaff3cb4701a2f7b8cf8ae0520d26ee3ea5a04548e3f720725ee4e2c209d"} Sep 09 23:51:59.331990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:51:59.331969 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" event={"ID":"9524c44e-5e0f-45c7-bd16-7e086bee0370","Type":"ContainerStarted","Data":"857501886e588464942ef1105ed8a0d9fee8dd42269b8be38146a60a179cc296"} Sep 09 23:52:01.451906 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.451868 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs"] Sep 09 23:52:01.454512 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.454487 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" Sep 09 23:52:01.456894 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.456872 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lmtqt\"" Sep 09 23:52:01.461924 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.461897 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs"] Sep 09 23:52:01.512634 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.512596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvsqh\" (UniqueName: \"kubernetes.io/projected/ecd5c94f-0047-4a53-ae70-9a6b50dbddf6-kube-api-access-mvsqh\") pod \"cert-manager-cainjector-559cdd8c87-jr2cs\" (UID: \"ecd5c94f-0047-4a53-ae70-9a6b50dbddf6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" Sep 09 23:52:01.512801 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.512650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecd5c94f-0047-4a53-ae70-9a6b50dbddf6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jr2cs\" (UID: \"ecd5c94f-0047-4a53-ae70-9a6b50dbddf6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" Sep 09 23:52:01.614077 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.614040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvsqh\" (UniqueName: \"kubernetes.io/projected/ecd5c94f-0047-4a53-ae70-9a6b50dbddf6-kube-api-access-mvsqh\") pod \"cert-manager-cainjector-559cdd8c87-jr2cs\" (UID: \"ecd5c94f-0047-4a53-ae70-9a6b50dbddf6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" Sep 09 23:52:01.614253 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.614094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecd5c94f-0047-4a53-ae70-9a6b50dbddf6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jr2cs\" (UID: \"ecd5c94f-0047-4a53-ae70-9a6b50dbddf6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" Sep 09 23:52:01.621893 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.621863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecd5c94f-0047-4a53-ae70-9a6b50dbddf6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jr2cs\" (UID: \"ecd5c94f-0047-4a53-ae70-9a6b50dbddf6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" Sep 09 23:52:01.622008 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.621950 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvsqh\" (UniqueName: \"kubernetes.io/projected/ecd5c94f-0047-4a53-ae70-9a6b50dbddf6-kube-api-access-mvsqh\") pod \"cert-manager-cainjector-559cdd8c87-jr2cs\" (UID: \"ecd5c94f-0047-4a53-ae70-9a6b50dbddf6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" Sep 09 23:52:01.765359 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:01.765270 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" Sep 09 23:52:02.530661 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:02.530640 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs"] Sep 09 23:52:02.532707 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:52:02.532681 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecd5c94f_0047_4a53_ae70_9a6b50dbddf6.slice/crio-9edb9aa24029c83f8fe5e6c08d57961e3eeb1b404de1a9641a5e0cb383e88987 WatchSource:0}: Error finding container 9edb9aa24029c83f8fe5e6c08d57961e3eeb1b404de1a9641a5e0cb383e88987: Status 404 returned error can't find the container with id 9edb9aa24029c83f8fe5e6c08d57961e3eeb1b404de1a9641a5e0cb383e88987 Sep 09 23:52:03.351697 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:03.351666 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerID="35fa4927623132ba159641fbd36bb6c13f956b4d30c3648348a4a17ea2c6fc14" exitCode=0 Sep 09 23:52:03.351901 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:03.351768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" event={"ID":"8d2c2841-234c-4d20-b6d9-a9259966e089","Type":"ContainerDied","Data":"35fa4927623132ba159641fbd36bb6c13f956b4d30c3648348a4a17ea2c6fc14"} Sep 09 23:52:03.353542 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:03.353515 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" event={"ID":"9524c44e-5e0f-45c7-bd16-7e086bee0370","Type":"ContainerStarted","Data":"6693cd7f8da7a189fdcfef41f81704a8df288f8d565b517b9521af3ee12eab41"} Sep 09 23:52:03.353634 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:03.353562 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:52:03.355247 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:03.355227 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" event={"ID":"ecd5c94f-0047-4a53-ae70-9a6b50dbddf6","Type":"ContainerStarted","Data":"ade8dd984ca42e9797d13fac1ddb7815087be4492c78f8f7fe38fbd37c036fd3"} Sep 09 23:52:03.355349 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:03.355261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" event={"ID":"ecd5c94f-0047-4a53-ae70-9a6b50dbddf6","Type":"ContainerStarted","Data":"9edb9aa24029c83f8fe5e6c08d57961e3eeb1b404de1a9641a5e0cb383e88987"} Sep 09 23:52:03.381369 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:03.381326 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jr2cs" podStartSLOduration=2.381308827 podStartE2EDuration="2.381308827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:52:03.380889372 +0000 UTC m=+235.588029143" watchObservedRunningTime="2026-09-09 23:52:03.381308827 +0000 UTC m=+235.588448597" Sep 09 23:52:03.399083 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:03.399030 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" podStartSLOduration=2.113985634 podStartE2EDuration="5.399005516s" totalImagesPullingTime="3.285019882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:52:03.397388428 +0000 UTC m=+235.604528197" watchObservedRunningTime="2026-09-09 23:52:03.399005516 +0000 UTC m=+235.606145285" Sep 09 23:52:04.361304 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:04.361242 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerID="e13de2a88972059595799a213623e29ebcd4ae869283f8011dc66694c358ecef" exitCode=0 Sep 09 23:52:04.361769 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:04.361327 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" event={"ID":"8d2c2841-234c-4d20-b6d9-a9259966e089","Type":"ContainerDied","Data":"e13de2a88972059595799a213623e29ebcd4ae869283f8011dc66694c358ecef"} Sep 09 23:52:05.490005 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.489976 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:52:05.549843 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.549816 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-util\") pod \"8d2c2841-234c-4d20-b6d9-a9259966e089\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " Sep 09 23:52:05.549987 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.549855 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-bundle\") pod \"8d2c2841-234c-4d20-b6d9-a9259966e089\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " Sep 09 23:52:05.549987 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.549880 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55t86\" (UniqueName: \"kubernetes.io/projected/8d2c2841-234c-4d20-b6d9-a9259966e089-kube-api-access-55t86\") pod \"8d2c2841-234c-4d20-b6d9-a9259966e089\" (UID: \"8d2c2841-234c-4d20-b6d9-a9259966e089\") " Sep 09 23:52:05.550327 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.550272 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-bundle" (OuterVolumeSpecName: "bundle") pod "8d2c2841-234c-4d20-b6d9-a9259966e089" (UID: "8d2c2841-234c-4d20-b6d9-a9259966e089"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:52:05.552022 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.552002 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2c2841-234c-4d20-b6d9-a9259966e089-kube-api-access-55t86" (OuterVolumeSpecName: "kube-api-access-55t86") pod "8d2c2841-234c-4d20-b6d9-a9259966e089" (UID: "8d2c2841-234c-4d20-b6d9-a9259966e089"). InnerVolumeSpecName "kube-api-access-55t86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:52:05.556271 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.556235 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-util" (OuterVolumeSpecName: "util") pod "8d2c2841-234c-4d20-b6d9-a9259966e089" (UID: "8d2c2841-234c-4d20-b6d9-a9259966e089"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:52:05.650602 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.650520 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-util\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:52:05.650602 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.650551 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d2c2841-234c-4d20-b6d9-a9259966e089-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:52:05.650602 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:05.650564 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55t86\" (UniqueName: \"kubernetes.io/projected/8d2c2841-234c-4d20-b6d9-a9259966e089-kube-api-access-55t86\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:52:06.371012 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:06.370979 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" event={"ID":"8d2c2841-234c-4d20-b6d9-a9259966e089","Type":"ContainerDied","Data":"b4e8aaff3cb4701a2f7b8cf8ae0520d26ee3ea5a04548e3f720725ee4e2c209d"} Sep 09 23:52:06.371202 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:06.371037 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4e8aaff3cb4701a2f7b8cf8ae0520d26ee3ea5a04548e3f720725ee4e2c209d" Sep 09 23:52:06.371202 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:06.370989 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9smr5" Sep 09 23:52:09.363467 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:09.363437 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zgkwz" Sep 09 23:52:15.568502 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.568466 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g"] Sep 09 23:52:15.568858 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.568722 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerName="util" Sep 09 23:52:15.568858 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.568733 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerName="util" Sep 09 23:52:15.568858 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.568742 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerName="pull" Sep 09 23:52:15.568858 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.568747 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerName="pull" Sep 09 23:52:15.568858 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.568764 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerName="extract" Sep 09 23:52:15.568858 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.568770 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerName="extract" Sep 09 23:52:15.568858 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.568812 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d2c2841-234c-4d20-b6d9-a9259966e089" containerName="extract" Sep 09 23:52:15.621291 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.621245 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g"] Sep 09 23:52:15.621439 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.621345 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" Sep 09 23:52:15.623968 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.623946 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 09 23:52:15.624642 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.624624 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9kn6n\"" Sep 09 23:52:15.624715 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.624642 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:52:15.728183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.728146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pz22\" (UniqueName: \"kubernetes.io/projected/2428e147-b977-462c-a13a-9099c45f03b0-kube-api-access-5pz22\") pod \"openshift-lws-operator-fd8ccff4c-d825g\" (UID: \"2428e147-b977-462c-a13a-9099c45f03b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" Sep 09 23:52:15.728377 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.728197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2428e147-b977-462c-a13a-9099c45f03b0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d825g\" (UID: \"2428e147-b977-462c-a13a-9099c45f03b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" Sep 09 23:52:15.829488 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.829403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pz22\" (UniqueName: \"kubernetes.io/projected/2428e147-b977-462c-a13a-9099c45f03b0-kube-api-access-5pz22\") pod \"openshift-lws-operator-fd8ccff4c-d825g\" (UID: \"2428e147-b977-462c-a13a-9099c45f03b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" Sep 09 23:52:15.829488 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.829451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2428e147-b977-462c-a13a-9099c45f03b0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d825g\" (UID: \"2428e147-b977-462c-a13a-9099c45f03b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" Sep 09 23:52:15.829866 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.829848 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2428e147-b977-462c-a13a-9099c45f03b0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d825g\" (UID: \"2428e147-b977-462c-a13a-9099c45f03b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" Sep 09 23:52:15.837862 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.837835 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pz22\" (UniqueName: \"kubernetes.io/projected/2428e147-b977-462c-a13a-9099c45f03b0-kube-api-access-5pz22\") pod \"openshift-lws-operator-fd8ccff4c-d825g\" (UID: \"2428e147-b977-462c-a13a-9099c45f03b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" Sep 09 23:52:15.930393 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:15.930359 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" Sep 09 23:52:16.213561 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:16.213542 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g"] Sep 09 23:52:16.216048 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:52:16.216021 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2428e147_b977_462c_a13a_9099c45f03b0.slice/crio-69f1877f3fe1d938a638068f17d6c11f760f97f970cdebb3d39d1f2538d548a1 WatchSource:0}: Error finding container 69f1877f3fe1d938a638068f17d6c11f760f97f970cdebb3d39d1f2538d548a1: Status 404 returned error can't find the container with id 69f1877f3fe1d938a638068f17d6c11f760f97f970cdebb3d39d1f2538d548a1 Sep 09 23:52:16.410965 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:16.410929 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" event={"ID":"2428e147-b977-462c-a13a-9099c45f03b0","Type":"ContainerStarted","Data":"69f1877f3fe1d938a638068f17d6c11f760f97f970cdebb3d39d1f2538d548a1"} Sep 09 23:52:19.424301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:19.424259 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" event={"ID":"2428e147-b977-462c-a13a-9099c45f03b0","Type":"ContainerStarted","Data":"dc843b49c6f88e20c9acdaf62550775d06dbca7697f2f569fe25d0222c94267e"} Sep 09 23:52:19.439237 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:19.439187 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d825g" podStartSLOduration=2.246677967 podStartE2EDuration="4.43917222s" totalImagesPullingTime="2.192494253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:52:19.438017824 +0000 UTC m=+251.645157594" watchObservedRunningTime="2026-09-09 23:52:19.43917222 +0000 UTC m=+251.646311999" Sep 09 23:52:22.720594 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.720554 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf"] Sep 09 23:52:22.722967 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.722949 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:22.725157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.725131 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-67ntj\"" Sep 09 23:52:22.725157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.725147 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 23:52:22.725344 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.725132 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 23:52:22.731153 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.731126 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf"] Sep 09 23:52:22.785002 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.784975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:22.785175 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.785027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-kube-api-access-h9rl8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:22.785175 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.785058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:22.886332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.886266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:22.886522 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.886343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-kube-api-access-h9rl8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:22.886522 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.886366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:22.886738 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.886713 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:22.886738 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.886730 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:22.894292 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:22.894255 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-kube-api-access-h9rl8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:23.032517 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:23.032419 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:23.301367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:23.301336 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf"] Sep 09 23:52:23.304132 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:52:23.304105 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb0e13e_5a70_468e_8e3b_a9a5c32728cc.slice/crio-b0cd8bce5f6fcab750f241d57637c076626e8c7c9c4362fbd1ff6b06de934be9 WatchSource:0}: Error finding container b0cd8bce5f6fcab750f241d57637c076626e8c7c9c4362fbd1ff6b06de934be9: Status 404 returned error can't find the container with id b0cd8bce5f6fcab750f241d57637c076626e8c7c9c4362fbd1ff6b06de934be9 Sep 09 23:52:23.442072 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:23.442036 2574 generic.go:356] "Generic (PLEG): container finished" podID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerID="ea51b43bc54806d07e7274cf6c45fbf7719765c0a7dc10599178af72ae73815b" exitCode=0 Sep 09 23:52:23.442207 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:23.442125 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" event={"ID":"adb0e13e-5a70-468e-8e3b-a9a5c32728cc","Type":"ContainerDied","Data":"ea51b43bc54806d07e7274cf6c45fbf7719765c0a7dc10599178af72ae73815b"} Sep 09 23:52:23.442207 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:23.442167 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" event={"ID":"adb0e13e-5a70-468e-8e3b-a9a5c32728cc","Type":"ContainerStarted","Data":"b0cd8bce5f6fcab750f241d57637c076626e8c7c9c4362fbd1ff6b06de934be9"} Sep 09 23:52:24.447892 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:24.447860 2574 generic.go:356] "Generic (PLEG): container finished" podID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerID="73ae265f6b15ae98bf85334cc81ba149134e3f9b4d11d65fdcd5a54708cb4ef5" exitCode=0 Sep 09 23:52:24.448384 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:24.447916 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" event={"ID":"adb0e13e-5a70-468e-8e3b-a9a5c32728cc","Type":"ContainerDied","Data":"73ae265f6b15ae98bf85334cc81ba149134e3f9b4d11d65fdcd5a54708cb4ef5"} Sep 09 23:52:25.454268 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:25.454230 2574 generic.go:356] "Generic (PLEG): container finished" podID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerID="28e9923ac328844d3871590fb0a67d9ed43c12add85e8038428046948855e2f3" exitCode=0 Sep 09 23:52:25.454671 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:25.454314 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" event={"ID":"adb0e13e-5a70-468e-8e3b-a9a5c32728cc","Type":"ContainerDied","Data":"28e9923ac328844d3871590fb0a67d9ed43c12add85e8038428046948855e2f3"} Sep 09 23:52:26.579083 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.579063 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:26.719397 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.719316 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-util\") pod \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " Sep 09 23:52:26.719397 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.719367 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-kube-api-access-h9rl8\") pod \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " Sep 09 23:52:26.719397 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.719394 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-bundle\") pod \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\" (UID: \"adb0e13e-5a70-468e-8e3b-a9a5c32728cc\") " Sep 09 23:52:26.720155 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.720124 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-bundle" (OuterVolumeSpecName: "bundle") pod "adb0e13e-5a70-468e-8e3b-a9a5c32728cc" (UID: "adb0e13e-5a70-468e-8e3b-a9a5c32728cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:52:26.721427 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.721403 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-kube-api-access-h9rl8" (OuterVolumeSpecName: "kube-api-access-h9rl8") pod "adb0e13e-5a70-468e-8e3b-a9a5c32728cc" (UID: "adb0e13e-5a70-468e-8e3b-a9a5c32728cc"). InnerVolumeSpecName "kube-api-access-h9rl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:52:26.725003 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.724983 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-util" (OuterVolumeSpecName: "util") pod "adb0e13e-5a70-468e-8e3b-a9a5c32728cc" (UID: "adb0e13e-5a70-468e-8e3b-a9a5c32728cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:52:26.820889 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.820843 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-util\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:52:26.820889 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.820885 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-kube-api-access-h9rl8\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:52:26.820889 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:26.820899 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb0e13e-5a70-468e-8e3b-a9a5c32728cc-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:52:27.464893 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:27.464861 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" Sep 09 23:52:27.464893 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:27.464880 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af48tcf" event={"ID":"adb0e13e-5a70-468e-8e3b-a9a5c32728cc","Type":"ContainerDied","Data":"b0cd8bce5f6fcab750f241d57637c076626e8c7c9c4362fbd1ff6b06de934be9"} Sep 09 23:52:27.465120 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:27.464933 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0cd8bce5f6fcab750f241d57637c076626e8c7c9c4362fbd1ff6b06de934be9" Sep 09 23:52:33.131499 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.131423 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6"] Sep 09 23:52:33.131851 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.131685 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerName="util" Sep 09 23:52:33.131851 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.131695 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerName="util" Sep 09 23:52:33.131851 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.131713 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerName="pull" Sep 09 23:52:33.131851 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.131720 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerName="pull" Sep 09 23:52:33.131851 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.131726 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerName="extract" Sep 09 23:52:33.131851 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.131732 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerName="extract" Sep 09 23:52:33.131851 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.131774 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb0e13e-5a70-468e-8e3b-a9a5c32728cc" containerName="extract" Sep 09 23:52:33.133814 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.133799 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.136176 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.136152 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 23:52:33.136340 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.136155 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 23:52:33.136340 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.136190 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-67ntj\"" Sep 09 23:52:33.144259 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.144236 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6"] Sep 09 23:52:33.268645 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.268607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.268821 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.268668 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.268821 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.268728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4plb\" (UniqueName: \"kubernetes.io/projected/052054b1-7d22-427f-ae16-3c64fb53e79e-kube-api-access-g4plb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.369829 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.369791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4plb\" (UniqueName: \"kubernetes.io/projected/052054b1-7d22-427f-ae16-3c64fb53e79e-kube-api-access-g4plb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.370007 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.369848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.370007 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.369899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.370279 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.370259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.370383 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.370316 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.386857 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.386802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4plb\" (UniqueName: \"kubernetes.io/projected/052054b1-7d22-427f-ae16-3c64fb53e79e-kube-api-access-g4plb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.442788 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.442754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:33.716774 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:33.716748 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6"] Sep 09 23:52:33.718790 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:52:33.718762 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052054b1_7d22_427f_ae16_3c64fb53e79e.slice/crio-04a936ffd0c7ef6a8181cbdd12bf4f2ade3bd440bb88fd231796deee5673b334 WatchSource:0}: Error finding container 04a936ffd0c7ef6a8181cbdd12bf4f2ade3bd440bb88fd231796deee5673b334: Status 404 returned error can't find the container with id 04a936ffd0c7ef6a8181cbdd12bf4f2ade3bd440bb88fd231796deee5673b334 Sep 09 23:52:34.496595 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.496564 2574 generic.go:356] "Generic (PLEG): container finished" podID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerID="85a01a34bf779ed3ebc88da701721a76d74ed84107ec8174c5f551ddd12a6936" exitCode=0 Sep 09 23:52:34.497008 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.496616 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" event={"ID":"052054b1-7d22-427f-ae16-3c64fb53e79e","Type":"ContainerDied","Data":"85a01a34bf779ed3ebc88da701721a76d74ed84107ec8174c5f551ddd12a6936"} Sep 09 23:52:34.497008 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.496652 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" event={"ID":"052054b1-7d22-427f-ae16-3c64fb53e79e","Type":"ContainerStarted","Data":"04a936ffd0c7ef6a8181cbdd12bf4f2ade3bd440bb88fd231796deee5673b334"} Sep 09 23:52:34.742717 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.742644 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp"] Sep 09 23:52:34.744871 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.744856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.747597 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.747573 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 23:52:34.747720 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.747601 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 23:52:34.747720 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.747574 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hsxhq\"" Sep 09 23:52:34.747995 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.747981 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 23:52:34.748347 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.748331 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 23:52:34.759685 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.759663 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp"] Sep 09 23:52:34.879659 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.879626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.879813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.879667 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-manifests\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.879813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.879692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-webhook-cert\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.879813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.879767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcvc7\" (UniqueName: \"kubernetes.io/projected/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-kube-api-access-tcvc7\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.879910 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.879832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-tmp\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.980536 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.980490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.980704 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.980565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-manifests\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.980704 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.980603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-webhook-cert\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.980704 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.980645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcvc7\" (UniqueName: \"kubernetes.io/projected/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-kube-api-access-tcvc7\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.980704 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.980690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-tmp\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.981043 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.981012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-manifests\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.982781 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.982760 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-tmp\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.982880 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.982853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-webhook-cert\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.982959 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.982942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:34.988540 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:34.988517 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcvc7\" (UniqueName: \"kubernetes.io/projected/2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545-kube-api-access-tcvc7\") pod \"opendatahub-operator-controller-manager-7547cdb9fd-hdptp\" (UID: \"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545\") " pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:35.054493 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:35.054473 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:35.328117 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:35.328081 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp"] Sep 09 23:52:35.330797 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:52:35.330764 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdb9a5b_c5f8_4ed9_bc4d_6acf27dda545.slice/crio-ccc7db9f2ef94e595a6cd3ec19235d442283bcb34a94e48aa0ab561ca16d823a WatchSource:0}: Error finding container ccc7db9f2ef94e595a6cd3ec19235d442283bcb34a94e48aa0ab561ca16d823a: Status 404 returned error can't find the container with id ccc7db9f2ef94e595a6cd3ec19235d442283bcb34a94e48aa0ab561ca16d823a Sep 09 23:52:35.507924 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:35.507884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" event={"ID":"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545","Type":"ContainerStarted","Data":"ccc7db9f2ef94e595a6cd3ec19235d442283bcb34a94e48aa0ab561ca16d823a"} Sep 09 23:52:35.509606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:35.509581 2574 generic.go:356] "Generic (PLEG): container finished" podID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerID="f49a11444a2f876fc0f7108b6572bac0da2ed841679ca360fa3fc2fde805f53d" exitCode=0 Sep 09 23:52:35.509710 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:35.509612 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" event={"ID":"052054b1-7d22-427f-ae16-3c64fb53e79e","Type":"ContainerDied","Data":"f49a11444a2f876fc0f7108b6572bac0da2ed841679ca360fa3fc2fde805f53d"} Sep 09 23:52:36.520510 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:36.520318 2574 generic.go:356] "Generic (PLEG): container finished" podID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerID="7df88df37f1f24dd1db5aef24ecd9224d323164e0732bfa0554f4d0686e382a8" exitCode=0 Sep 09 23:52:36.520510 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:36.520443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" event={"ID":"052054b1-7d22-427f-ae16-3c64fb53e79e","Type":"ContainerDied","Data":"7df88df37f1f24dd1db5aef24ecd9224d323164e0732bfa0554f4d0686e382a8"} Sep 09 23:52:38.398316 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.398274 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:38.511606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.511581 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-util\") pod \"052054b1-7d22-427f-ae16-3c64fb53e79e\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " Sep 09 23:52:38.511710 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.511621 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4plb\" (UniqueName: \"kubernetes.io/projected/052054b1-7d22-427f-ae16-3c64fb53e79e-kube-api-access-g4plb\") pod \"052054b1-7d22-427f-ae16-3c64fb53e79e\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " Sep 09 23:52:38.511710 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.511647 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-bundle\") pod \"052054b1-7d22-427f-ae16-3c64fb53e79e\" (UID: \"052054b1-7d22-427f-ae16-3c64fb53e79e\") " Sep 09 23:52:38.512345 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.512320 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-bundle" (OuterVolumeSpecName: "bundle") pod "052054b1-7d22-427f-ae16-3c64fb53e79e" (UID: "052054b1-7d22-427f-ae16-3c64fb53e79e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:52:38.513462 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.513435 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052054b1-7d22-427f-ae16-3c64fb53e79e-kube-api-access-g4plb" (OuterVolumeSpecName: "kube-api-access-g4plb") pod "052054b1-7d22-427f-ae16-3c64fb53e79e" (UID: "052054b1-7d22-427f-ae16-3c64fb53e79e"). InnerVolumeSpecName "kube-api-access-g4plb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:52:38.517202 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.517164 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-util" (OuterVolumeSpecName: "util") pod "052054b1-7d22-427f-ae16-3c64fb53e79e" (UID: "052054b1-7d22-427f-ae16-3c64fb53e79e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:52:38.534538 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.534522 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" Sep 09 23:52:38.534627 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.534535 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0vz2h6" event={"ID":"052054b1-7d22-427f-ae16-3c64fb53e79e","Type":"ContainerDied","Data":"04a936ffd0c7ef6a8181cbdd12bf4f2ade3bd440bb88fd231796deee5673b334"} Sep 09 23:52:38.534627 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.534574 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04a936ffd0c7ef6a8181cbdd12bf4f2ade3bd440bb88fd231796deee5673b334" Sep 09 23:52:38.536183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.536158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" event={"ID":"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545","Type":"ContainerStarted","Data":"e3cbe8833e25464256844572f77df434a4a9e53491ae2cbbff6765b36df69b23"} Sep 09 23:52:38.613020 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.612997 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-util\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:52:38.613020 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.613019 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4plb\" (UniqueName: \"kubernetes.io/projected/052054b1-7d22-427f-ae16-3c64fb53e79e-kube-api-access-g4plb\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:52:38.613151 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:38.613030 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/052054b1-7d22-427f-ae16-3c64fb53e79e-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:52:39.542565 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:39.542540 2574 generic.go:356] "Generic (PLEG): container finished" podID="2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545" containerID="e3cbe8833e25464256844572f77df434a4a9e53491ae2cbbff6765b36df69b23" exitCode=0 Sep 09 23:52:39.542906 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:39.542577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" event={"ID":"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545","Type":"ContainerDied","Data":"e3cbe8833e25464256844572f77df434a4a9e53491ae2cbbff6765b36df69b23"} Sep 09 23:52:40.549964 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:40.549928 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" event={"ID":"2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545","Type":"ContainerStarted","Data":"e4d0aefd5cee593dd2c8a8cdf4ea3cc2363fa4a6ae8adcf3d75ee272a38f97dc"} Sep 09 23:52:40.550457 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:40.550102 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:40.571225 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:40.571176 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" podStartSLOduration=2.101469717 podStartE2EDuration="6.571162378s" totalImagesPullingTime="3.424439565s" totalInitContainerRuntime="1.045253096s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:52:40.570690343 +0000 UTC m=+272.777830113" watchObservedRunningTime="2026-09-09 23:52:40.571162378 +0000 UTC m=+272.778302147" Sep 09 23:52:51.556806 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:51.556771 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7547cdb9fd-hdptp" Sep 09 23:52:54.048707 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.048666 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-947759788-x4q2t"] Sep 09 23:52:54.049179 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.049092 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerName="pull" Sep 09 23:52:54.049179 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.049111 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerName="pull" Sep 09 23:52:54.049179 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.049138 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerName="util" Sep 09 23:52:54.049179 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.049146 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerName="util" Sep 09 23:52:54.049179 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.049156 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerName="extract" Sep 09 23:52:54.049179 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.049164 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerName="extract" Sep 09 23:52:54.049496 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.049252 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="052054b1-7d22-427f-ae16-3c64fb53e79e" containerName="extract" Sep 09 23:52:54.059419 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.059393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.061813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.061785 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 23:52:54.061813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.061795 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 09 23:52:54.062041 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.061846 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 23:52:54.062041 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.061910 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 09 23:52:54.065917 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.065892 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-947759788-x4q2t"] Sep 09 23:52:54.122923 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.122890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e78d58b9-4f18-4f8a-bf6e-7c86427779f2-tls-certs\") pod \"kube-auth-proxy-947759788-x4q2t\" (UID: \"e78d58b9-4f18-4f8a-bf6e-7c86427779f2\") " pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.122923 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.122924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwr85\" (UniqueName: \"kubernetes.io/projected/e78d58b9-4f18-4f8a-bf6e-7c86427779f2-kube-api-access-rwr85\") pod \"kube-auth-proxy-947759788-x4q2t\" (UID: \"e78d58b9-4f18-4f8a-bf6e-7c86427779f2\") " pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.123127 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.122973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e78d58b9-4f18-4f8a-bf6e-7c86427779f2-tmp\") pod \"kube-auth-proxy-947759788-x4q2t\" (UID: \"e78d58b9-4f18-4f8a-bf6e-7c86427779f2\") " pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.224260 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.224223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e78d58b9-4f18-4f8a-bf6e-7c86427779f2-tls-certs\") pod \"kube-auth-proxy-947759788-x4q2t\" (UID: \"e78d58b9-4f18-4f8a-bf6e-7c86427779f2\") " pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.224425 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.224266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwr85\" (UniqueName: \"kubernetes.io/projected/e78d58b9-4f18-4f8a-bf6e-7c86427779f2-kube-api-access-rwr85\") pod \"kube-auth-proxy-947759788-x4q2t\" (UID: \"e78d58b9-4f18-4f8a-bf6e-7c86427779f2\") " pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.224425 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.224317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e78d58b9-4f18-4f8a-bf6e-7c86427779f2-tmp\") pod \"kube-auth-proxy-947759788-x4q2t\" (UID: \"e78d58b9-4f18-4f8a-bf6e-7c86427779f2\") " pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.226494 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.226471 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e78d58b9-4f18-4f8a-bf6e-7c86427779f2-tmp\") pod \"kube-auth-proxy-947759788-x4q2t\" (UID: \"e78d58b9-4f18-4f8a-bf6e-7c86427779f2\") " pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.226752 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.226732 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e78d58b9-4f18-4f8a-bf6e-7c86427779f2-tls-certs\") pod \"kube-auth-proxy-947759788-x4q2t\" (UID: \"e78d58b9-4f18-4f8a-bf6e-7c86427779f2\") " pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.231437 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.231420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwr85\" (UniqueName: \"kubernetes.io/projected/e78d58b9-4f18-4f8a-bf6e-7c86427779f2-kube-api-access-rwr85\") pod \"kube-auth-proxy-947759788-x4q2t\" (UID: \"e78d58b9-4f18-4f8a-bf6e-7c86427779f2\") " pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.370820 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.370752 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" Sep 09 23:52:54.642833 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:54.642652 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-947759788-x4q2t"] Sep 09 23:52:54.645375 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:52:54.645340 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78d58b9_4f18_4f8a_bf6e_7c86427779f2.slice/crio-dfa73ac3ed7a3de0aec2750271b9f44b6110791379c003db779d6f7255a1f47e WatchSource:0}: Error finding container dfa73ac3ed7a3de0aec2750271b9f44b6110791379c003db779d6f7255a1f47e: Status 404 returned error can't find the container with id dfa73ac3ed7a3de0aec2750271b9f44b6110791379c003db779d6f7255a1f47e Sep 09 23:52:55.256609 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.256578 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq"] Sep 09 23:52:55.258861 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.258842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.261426 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.261367 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 23:52:55.261426 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.261396 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5zmbg\"" Sep 09 23:52:55.261426 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.261422 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 09 23:52:55.261665 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.261401 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 09 23:52:55.261665 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.261422 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 09 23:52:55.276395 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.276375 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq"] Sep 09 23:52:55.334126 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.334099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.334313 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.334169 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.334313 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.334212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.334313 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.334234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.334313 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.334265 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc07135-7071-4179-9fdb-f37d19238854-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.334534 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.334332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.334534 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.334356 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n58qq\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-kube-api-access-n58qq\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.434754 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.434720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.434928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.434778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.434928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.434806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.434928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.434834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc07135-7071-4179-9fdb-f37d19238854-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.434928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.434876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.434928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.434899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n58qq\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-kube-api-access-n58qq\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.435173 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.434930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.435819 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.435786 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.437775 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.437753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.437877 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.437857 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.437947 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.437917 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.438209 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.438170 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc07135-7071-4179-9fdb-f37d19238854-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.442945 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.442926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.443366 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.443345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n58qq\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-kube-api-access-n58qq\") pod \"istiod-openshift-gateway-7b7995d5cd-pxhdq\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.571192 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.571151 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:55.618537 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.618497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" event={"ID":"e78d58b9-4f18-4f8a-bf6e-7c86427779f2","Type":"ContainerStarted","Data":"dfa73ac3ed7a3de0aec2750271b9f44b6110791379c003db779d6f7255a1f47e"} Sep 09 23:52:55.853516 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:55.853416 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq"] Sep 09 23:52:56.625485 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:56.625450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" event={"ID":"bfc07135-7071-4179-9fdb-f37d19238854","Type":"ContainerStarted","Data":"df9cdd8178b4ff813f59c6c4911e0c60f75995291855641b16072d90c54d1b7e"} Sep 09 23:52:57.633472 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:57.633181 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" event={"ID":"e78d58b9-4f18-4f8a-bf6e-7c86427779f2","Type":"ContainerStarted","Data":"693a7b555b7d5a30e6407aa114dbdca28a8692de5be4fadc03d2cf84655ec558"} Sep 09 23:52:57.652119 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:57.652063 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-947759788-x4q2t" podStartSLOduration=0.862142762 podStartE2EDuration="3.65204681s" totalImagesPullingTime="2.789904048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:52:57.649165282 +0000 UTC m=+289.856305043" watchObservedRunningTime="2026-09-09 23:52:57.65204681 +0000 UTC m=+289.859186581" Sep 09 23:52:59.213760 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:59.213721 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 09 23:52:59.214056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:59.213789 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 09 23:52:59.644449 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:59.644421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" event={"ID":"bfc07135-7071-4179-9fdb-f37d19238854","Type":"ContainerStarted","Data":"00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7"} Sep 09 23:52:59.644631 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:59.644519 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:52:59.680191 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:52:59.680138 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" podStartSLOduration=1.32744124 podStartE2EDuration="4.680124738s" totalImagesPullingTime="3.352683498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:52:59.678632417 +0000 UTC m=+291.885772189" watchObservedRunningTime="2026-09-09 23:52:59.680124738 +0000 UTC m=+291.887264507" Sep 09 23:53:00.651058 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:00.651027 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:53:08.162321 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:08.162262 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 09 23:53:08.165543 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:08.165518 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 09 23:53:08.171546 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:08.171529 2574 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 23:53:13.959530 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.959492 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pkslh"] Sep 09 23:53:13.962888 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.962870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:13.965888 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.965843 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gcvdh\"" Sep 09 23:53:13.967713 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.967695 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 09 23:53:13.975351 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.974014 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pkslh"] Sep 09 23:53:13.982513 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.982492 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546"] Sep 09 23:53:13.985560 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.985542 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:13.988377 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.988350 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 09 23:53:13.988465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.988417 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kncfd\"" Sep 09 23:53:13.999781 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:13.999760 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546"] Sep 09 23:53:14.091636 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.091612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkslh\" (UID: \"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:14.091766 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.091648 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxzc\" (UniqueName: \"kubernetes.io/projected/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-kube-api-access-vbxzc\") pod \"kserve-controller-manager-d6f4b65b8-pkslh\" (UID: \"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:14.091766 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.091667 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fce0520e-d5e7-4e3f-9532-a1765adffbad-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jm546\" (UID: \"fce0520e-d5e7-4e3f-9532-a1765adffbad\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:14.091766 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.091737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6frq2\" (UniqueName: \"kubernetes.io/projected/fce0520e-d5e7-4e3f-9532-a1765adffbad-kube-api-access-6frq2\") pod \"llmisvc-controller-manager-777c6bd7d5-jm546\" (UID: \"fce0520e-d5e7-4e3f-9532-a1765adffbad\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:14.195031 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.194997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6frq2\" (UniqueName: \"kubernetes.io/projected/fce0520e-d5e7-4e3f-9532-a1765adffbad-kube-api-access-6frq2\") pod \"llmisvc-controller-manager-777c6bd7d5-jm546\" (UID: \"fce0520e-d5e7-4e3f-9532-a1765adffbad\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:14.195254 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.195089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkslh\" (UID: \"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:14.195254 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.195133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxzc\" (UniqueName: \"kubernetes.io/projected/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-kube-api-access-vbxzc\") pod \"kserve-controller-manager-d6f4b65b8-pkslh\" (UID: \"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:14.195254 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.195161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fce0520e-d5e7-4e3f-9532-a1765adffbad-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jm546\" (UID: \"fce0520e-d5e7-4e3f-9532-a1765adffbad\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:14.195501 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:53:14.195265 2574 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 09 23:53:14.195501 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:53:14.195358 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-cert podName:25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4 nodeName:}" failed. No retries permitted until 2026-09-09 23:53:14.695336715 +0000 UTC m=+306.902476467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-cert") pod "kserve-controller-manager-d6f4b65b8-pkslh" (UID: "25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4") : secret "kserve-webhook-server-cert" not found Sep 09 23:53:14.197660 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.197640 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fce0520e-d5e7-4e3f-9532-a1765adffbad-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jm546\" (UID: \"fce0520e-d5e7-4e3f-9532-a1765adffbad\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:14.216085 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.216032 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6frq2\" (UniqueName: \"kubernetes.io/projected/fce0520e-d5e7-4e3f-9532-a1765adffbad-kube-api-access-6frq2\") pod \"llmisvc-controller-manager-777c6bd7d5-jm546\" (UID: \"fce0520e-d5e7-4e3f-9532-a1765adffbad\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:14.216375 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.216357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxzc\" (UniqueName: \"kubernetes.io/projected/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-kube-api-access-vbxzc\") pod \"kserve-controller-manager-d6f4b65b8-pkslh\" (UID: \"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:14.294720 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.294695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:14.565295 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.565260 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546"] Sep 09 23:53:14.567052 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:53:14.567024 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfce0520e_d5e7_4e3f_9532_a1765adffbad.slice/crio-8d130204226deec7b852bb3aaefa2cdc7f17fa038bb52205bc553f03f9a8a96b WatchSource:0}: Error finding container 8d130204226deec7b852bb3aaefa2cdc7f17fa038bb52205bc553f03f9a8a96b: Status 404 returned error can't find the container with id 8d130204226deec7b852bb3aaefa2cdc7f17fa038bb52205bc553f03f9a8a96b Sep 09 23:53:14.568445 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.568427 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:53:14.699194 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.699163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkslh\" (UID: \"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:14.699408 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:53:14.699365 2574 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 09 23:53:14.699486 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:53:14.699476 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-cert podName:25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4 nodeName:}" failed. No retries permitted until 2026-09-09 23:53:15.699453222 +0000 UTC m=+307.906592985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-cert") pod "kserve-controller-manager-d6f4b65b8-pkslh" (UID: "25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4") : secret "kserve-webhook-server-cert" not found Sep 09 23:53:14.713743 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:14.713715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" event={"ID":"fce0520e-d5e7-4e3f-9532-a1765adffbad","Type":"ContainerStarted","Data":"8d130204226deec7b852bb3aaefa2cdc7f17fa038bb52205bc553f03f9a8a96b"} Sep 09 23:53:15.708534 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:15.708496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkslh\" (UID: \"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:15.711017 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:15.710990 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkslh\" (UID: \"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:15.774458 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:15.774419 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:15.995262 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:15.995192 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq"] Sep 09 23:53:15.995466 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:15.995440 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" podUID="bfc07135-7071-4179-9fdb-f37d19238854" containerName="discovery" containerID="cri-o://00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7" gracePeriod=30 Sep 09 23:53:16.163239 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.163195 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pkslh"] Sep 09 23:53:16.165418 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:53:16.165389 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d4eb97_dc5e_456f_bf97_d20ec3b6c0a4.slice/crio-ea15599064b1a8dd9368af08f3a044c6efefbdbf056ffa5cae6536b0d37d23c3 WatchSource:0}: Error finding container ea15599064b1a8dd9368af08f3a044c6efefbdbf056ffa5cae6536b0d37d23c3: Status 404 returned error can't find the container with id ea15599064b1a8dd9368af08f3a044c6efefbdbf056ffa5cae6536b0d37d23c3 Sep 09 23:53:16.246930 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.246873 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:53:16.315839 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.315810 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-istio-token\") pod \"bfc07135-7071-4179-9fdb-f37d19238854\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " Sep 09 23:53:16.315983 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.315877 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n58qq\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-kube-api-access-n58qq\") pod \"bfc07135-7071-4179-9fdb-f37d19238854\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " Sep 09 23:53:16.315983 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.315899 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-ca-configmap\") pod \"bfc07135-7071-4179-9fdb-f37d19238854\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " Sep 09 23:53:16.315983 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.315954 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-cacerts\") pod \"bfc07135-7071-4179-9fdb-f37d19238854\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " Sep 09 23:53:16.316125 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.316004 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-dns-cert\") pod \"bfc07135-7071-4179-9fdb-f37d19238854\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " Sep 09 23:53:16.316125 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.316046 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc07135-7071-4179-9fdb-f37d19238854-local-certs\") pod \"bfc07135-7071-4179-9fdb-f37d19238854\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " Sep 09 23:53:16.316125 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.316078 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-kubeconfig\") pod \"bfc07135-7071-4179-9fdb-f37d19238854\" (UID: \"bfc07135-7071-4179-9fdb-f37d19238854\") " Sep 09 23:53:16.316299 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.316254 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "bfc07135-7071-4179-9fdb-f37d19238854" (UID: "bfc07135-7071-4179-9fdb-f37d19238854"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:53:16.316422 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.316404 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-ca-configmap\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:53:16.318332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.318249 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "bfc07135-7071-4179-9fdb-f37d19238854" (UID: "bfc07135-7071-4179-9fdb-f37d19238854"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:53:16.318332 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.318267 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfc07135-7071-4179-9fdb-f37d19238854" (UID: "bfc07135-7071-4179-9fdb-f37d19238854"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:53:16.318502 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.318483 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-cacerts" (OuterVolumeSpecName: "cacerts") pod "bfc07135-7071-4179-9fdb-f37d19238854" (UID: "bfc07135-7071-4179-9fdb-f37d19238854"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:53:16.318558 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.318540 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc07135-7071-4179-9fdb-f37d19238854-local-certs" (OuterVolumeSpecName: "local-certs") pod "bfc07135-7071-4179-9fdb-f37d19238854" (UID: "bfc07135-7071-4179-9fdb-f37d19238854"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:53:16.318605 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.318552 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-kube-api-access-n58qq" (OuterVolumeSpecName: "kube-api-access-n58qq") pod "bfc07135-7071-4179-9fdb-f37d19238854" (UID: "bfc07135-7071-4179-9fdb-f37d19238854"). InnerVolumeSpecName "kube-api-access-n58qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:53:16.318651 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.318637 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "bfc07135-7071-4179-9fdb-f37d19238854" (UID: "bfc07135-7071-4179-9fdb-f37d19238854"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:53:16.417576 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.417547 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc07135-7071-4179-9fdb-f37d19238854-local-certs\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:53:16.417576 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.417574 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-kubeconfig\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:53:16.417747 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.417583 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-istio-token\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:53:16.417747 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.417592 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n58qq\" (UniqueName: \"kubernetes.io/projected/bfc07135-7071-4179-9fdb-f37d19238854-kube-api-access-n58qq\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:53:16.417747 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.417602 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-cacerts\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:53:16.417747 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.417610 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfc07135-7071-4179-9fdb-f37d19238854-istio-csr-dns-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:53:16.727945 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.727909 2574 generic.go:356] "Generic (PLEG): container finished" podID="bfc07135-7071-4179-9fdb-f37d19238854" containerID="00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7" exitCode=0 Sep 09 23:53:16.728433 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.727965 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" Sep 09 23:53:16.728433 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.727974 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" event={"ID":"bfc07135-7071-4179-9fdb-f37d19238854","Type":"ContainerDied","Data":"00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7"} Sep 09 23:53:16.728433 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.728033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq" event={"ID":"bfc07135-7071-4179-9fdb-f37d19238854","Type":"ContainerDied","Data":"df9cdd8178b4ff813f59c6c4911e0c60f75995291855641b16072d90c54d1b7e"} Sep 09 23:53:16.728433 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.728057 2574 scope.go:117] "RemoveContainer" containerID="00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7" Sep 09 23:53:16.730061 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.730026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" event={"ID":"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4","Type":"ContainerStarted","Data":"ea15599064b1a8dd9368af08f3a044c6efefbdbf056ffa5cae6536b0d37d23c3"} Sep 09 23:53:16.751197 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.751175 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq"] Sep 09 23:53:16.755600 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:16.755574 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-pxhdq"] Sep 09 23:53:17.011425 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:17.011176 2574 scope.go:117] "RemoveContainer" containerID="00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7" Sep 09 23:53:17.011641 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:53:17.011610 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7\": container with ID starting with 00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7 not found: ID does not exist" containerID="00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7" Sep 09 23:53:17.011765 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:17.011653 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7"} err="failed to get container status \"00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7\": rpc error: code = NotFound desc = could not find container \"00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7\": container with ID starting with 00003d357804f45845c963e43b3487a66de43f433ce85bede75ea53a88d12ec7 not found: ID does not exist" Sep 09 23:53:17.736813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:17.736776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" event={"ID":"fce0520e-d5e7-4e3f-9532-a1765adffbad","Type":"ContainerStarted","Data":"2bc992c3dab29fcf4945f565ec31a05d884a8879d793b0128bcd6a4a98831098"} Sep 09 23:53:17.737199 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:17.736900 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:17.755150 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:17.755095 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" podStartSLOduration=2.263806675 podStartE2EDuration="4.755080212s" totalImagesPullingTime="2.491273537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:17.752874731 +0000 UTC m=+309.960014494" watchObservedRunningTime="2026-09-09 23:53:17.755080212 +0000 UTC m=+309.962219982" Sep 09 23:53:18.239598 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:18.239569 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfc07135-7071-4179-9fdb-f37d19238854" path="/var/lib/kubelet/pods/bfc07135-7071-4179-9fdb-f37d19238854/volumes" Sep 09 23:53:19.747341 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:19.747307 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" event={"ID":"25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4","Type":"ContainerStarted","Data":"9ca43a8a73c12f92a84ccfc37ba8e09012b58083d57c7322fa4c8e928d42237c"} Sep 09 23:53:19.747685 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:19.747398 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:53:19.764397 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:19.764352 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" podStartSLOduration=4.152798661 podStartE2EDuration="6.764337634s" totalImagesPullingTime="2.611538973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:19.762303233 +0000 UTC m=+311.969442999" watchObservedRunningTime="2026-09-09 23:53:19.764337634 +0000 UTC m=+311.971477419" Sep 09 23:53:48.743557 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:48.743523 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jm546" Sep 09 23:53:50.757297 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:53:50.757252 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkslh" Sep 09 23:54:09.633716 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.633681 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c"] Sep 09 23:54:09.634182 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.634163 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc07135-7071-4179-9fdb-f37d19238854" containerName="discovery" Sep 09 23:54:09.634254 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.634185 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc07135-7071-4179-9fdb-f37d19238854" containerName="discovery" Sep 09 23:54:09.634338 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.634277 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfc07135-7071-4179-9fdb-f37d19238854" containerName="discovery" Sep 09 23:54:09.638265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.638243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.640681 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.640659 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 23:54:09.640841 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.640822 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 23:54:09.641491 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.641479 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-67ntj\"" Sep 09 23:54:09.645468 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.645443 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c"] Sep 09 23:54:09.722909 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.722880 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.723021 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.722912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxp8c\" (UniqueName: \"kubernetes.io/projected/32b9103a-e59f-49ad-8b4b-80c42c68fea5-kube-api-access-rxp8c\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.723021 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.722945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.729897 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.729873 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6"] Sep 09 23:54:09.732926 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.732912 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.741915 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.741895 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6"] Sep 09 23:54:09.824062 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.824029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.824215 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.824073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.824215 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.824115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.824215 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.824145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crrx\" (UniqueName: \"kubernetes.io/projected/b37a473a-3451-4ce8-bee6-9576a593f605-kube-api-access-7crrx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.824410 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.824239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.824410 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.824300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxp8c\" (UniqueName: \"kubernetes.io/projected/32b9103a-e59f-49ad-8b4b-80c42c68fea5-kube-api-access-rxp8c\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.824410 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.824404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.824567 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.824536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.828942 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.828919 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87"] Sep 09 23:54:09.832105 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.832089 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:09.836275 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.836255 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxp8c\" (UniqueName: \"kubernetes.io/projected/32b9103a-e59f-49ad-8b4b-80c42c68fea5-kube-api-access-rxp8c\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:09.839810 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.839791 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87"] Sep 09 23:54:09.925763 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.925686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:09.925763 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.925738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.925936 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.925809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.925936 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.925839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnv4\" (UniqueName: \"kubernetes.io/projected/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-kube-api-access-htnv4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:09.925936 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.925871 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crrx\" (UniqueName: \"kubernetes.io/projected/b37a473a-3451-4ce8-bee6-9576a593f605-kube-api-access-7crrx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.926062 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.925930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:09.926098 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.926065 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.926198 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.926175 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.932778 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.932747 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk"] Sep 09 23:54:09.934819 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.934795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crrx\" (UniqueName: \"kubernetes.io/projected/b37a473a-3451-4ce8-bee6-9576a593f605-kube-api-access-7crrx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:09.936140 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.936125 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:09.943616 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.943593 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk"] Sep 09 23:54:09.947533 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:09.947510 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:10.027005 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.026911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:10.027005 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.026971 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:10.027005 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.026997 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.027590 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.027045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnv4\" (UniqueName: \"kubernetes.io/projected/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-kube-api-access-htnv4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:10.027590 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.027063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.027590 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.027082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ns9\" (UniqueName: \"kubernetes.io/projected/abad69f7-2b89-4c51-811d-fbd9be538345-kube-api-access-x6ns9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.027590 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.027377 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:10.027590 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.027376 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:10.034895 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.034863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnv4\" (UniqueName: \"kubernetes.io/projected/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-kube-api-access-htnv4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:10.041662 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.041639 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:10.127584 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.127552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.127771 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.127615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.127771 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.127644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ns9\" (UniqueName: \"kubernetes.io/projected/abad69f7-2b89-4c51-811d-fbd9be538345-kube-api-access-x6ns9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.127993 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.127953 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.128087 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.127996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.135579 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.135554 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ns9\" (UniqueName: \"kubernetes.io/projected/abad69f7-2b89-4c51-811d-fbd9be538345-kube-api-access-x6ns9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.147367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.147340 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:10.224704 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.224675 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c"] Sep 09 23:54:10.227803 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:54:10.227769 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b9103a_e59f_49ad_8b4b_80c42c68fea5.slice/crio-d7a2b4d444b44db0d5681bbd553d4f3fd3fb73c2fbbcd3bd96d6865676d8fa82 WatchSource:0}: Error finding container d7a2b4d444b44db0d5681bbd553d4f3fd3fb73c2fbbcd3bd96d6865676d8fa82: Status 404 returned error can't find the container with id d7a2b4d444b44db0d5681bbd553d4f3fd3fb73c2fbbcd3bd96d6865676d8fa82 Sep 09 23:54:10.250626 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.250600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:10.319990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.317876 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6"] Sep 09 23:54:10.322530 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:54:10.322488 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37a473a_3451_4ce8_bee6_9576a593f605.slice/crio-81964b672f007dc9f0f03c2f8c7c6a662860acdc8d8316fc5eaf9c81992f5473 WatchSource:0}: Error finding container 81964b672f007dc9f0f03c2f8c7c6a662860acdc8d8316fc5eaf9c81992f5473: Status 404 returned error can't find the container with id 81964b672f007dc9f0f03c2f8c7c6a662860acdc8d8316fc5eaf9c81992f5473 Sep 09 23:54:10.421077 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.421057 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87"] Sep 09 23:54:10.422582 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:54:10.422558 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510c52cc_e02c_4d7b_a0e5_3cea4b6201eb.slice/crio-abc4c175240370ef3852a2a0aa56982e5a64b7fe6f1a2469b7ef9ece37232d7c WatchSource:0}: Error finding container abc4c175240370ef3852a2a0aa56982e5a64b7fe6f1a2469b7ef9ece37232d7c: Status 404 returned error can't find the container with id abc4c175240370ef3852a2a0aa56982e5a64b7fe6f1a2469b7ef9ece37232d7c Sep 09 23:54:10.526752 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.526731 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk"] Sep 09 23:54:10.565081 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:54:10.565052 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabad69f7_2b89_4c51_811d_fbd9be538345.slice/crio-090590bdcddca182293d0ac10e3ded2925d22c2e56c30ec42d0cc63b55de4f74 WatchSource:0}: Error finding container 090590bdcddca182293d0ac10e3ded2925d22c2e56c30ec42d0cc63b55de4f74: Status 404 returned error can't find the container with id 090590bdcddca182293d0ac10e3ded2925d22c2e56c30ec42d0cc63b55de4f74 Sep 09 23:54:10.974468 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.974428 2574 generic.go:356] "Generic (PLEG): container finished" podID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerID="133918f663d48e3f9eee6bfa180338f517441a5028dc17f17e6dc64ae94f32d2" exitCode=0 Sep 09 23:54:10.974891 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.974522 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" event={"ID":"32b9103a-e59f-49ad-8b4b-80c42c68fea5","Type":"ContainerDied","Data":"133918f663d48e3f9eee6bfa180338f517441a5028dc17f17e6dc64ae94f32d2"} Sep 09 23:54:10.974891 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.974574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" event={"ID":"32b9103a-e59f-49ad-8b4b-80c42c68fea5","Type":"ContainerStarted","Data":"d7a2b4d444b44db0d5681bbd553d4f3fd3fb73c2fbbcd3bd96d6865676d8fa82"} Sep 09 23:54:10.976127 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.976078 2574 generic.go:356] "Generic (PLEG): container finished" podID="abad69f7-2b89-4c51-811d-fbd9be538345" containerID="4ce56a8397e8f659ec317cbd39e0fde0b4390bca5135b644a2dd78e914b33be2" exitCode=0 Sep 09 23:54:10.976187 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.976162 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" event={"ID":"abad69f7-2b89-4c51-811d-fbd9be538345","Type":"ContainerDied","Data":"4ce56a8397e8f659ec317cbd39e0fde0b4390bca5135b644a2dd78e914b33be2"} Sep 09 23:54:10.976243 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.976213 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" event={"ID":"abad69f7-2b89-4c51-811d-fbd9be538345","Type":"ContainerStarted","Data":"090590bdcddca182293d0ac10e3ded2925d22c2e56c30ec42d0cc63b55de4f74"} Sep 09 23:54:10.977695 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.977672 2574 generic.go:356] "Generic (PLEG): container finished" podID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerID="3233442e42ba1d5cda6fc4b59d79d764e3f8b8bc6bb7e36b6764571f09e576ac" exitCode=0 Sep 09 23:54:10.977788 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.977756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" event={"ID":"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb","Type":"ContainerDied","Data":"3233442e42ba1d5cda6fc4b59d79d764e3f8b8bc6bb7e36b6764571f09e576ac"} Sep 09 23:54:10.977846 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.977798 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" event={"ID":"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb","Type":"ContainerStarted","Data":"abc4c175240370ef3852a2a0aa56982e5a64b7fe6f1a2469b7ef9ece37232d7c"} Sep 09 23:54:10.979467 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.979446 2574 generic.go:356] "Generic (PLEG): container finished" podID="b37a473a-3451-4ce8-bee6-9576a593f605" containerID="f87feee103e4b22aa5814b3f60f2d9d7ee012a7eddc4f6ec88aeaa114eb64a28" exitCode=0 Sep 09 23:54:10.979573 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.979477 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" event={"ID":"b37a473a-3451-4ce8-bee6-9576a593f605","Type":"ContainerDied","Data":"f87feee103e4b22aa5814b3f60f2d9d7ee012a7eddc4f6ec88aeaa114eb64a28"} Sep 09 23:54:10.979573 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:10.979531 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" event={"ID":"b37a473a-3451-4ce8-bee6-9576a593f605","Type":"ContainerStarted","Data":"81964b672f007dc9f0f03c2f8c7c6a662860acdc8d8316fc5eaf9c81992f5473"} Sep 09 23:54:11.985697 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:11.985638 2574 generic.go:356] "Generic (PLEG): container finished" podID="abad69f7-2b89-4c51-811d-fbd9be538345" containerID="c9d849e7121336ee074c0b043e787e92fdf711e242ecbeb902f250f19fa68300" exitCode=0 Sep 09 23:54:11.986056 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:11.985718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" event={"ID":"abad69f7-2b89-4c51-811d-fbd9be538345","Type":"ContainerDied","Data":"c9d849e7121336ee074c0b043e787e92fdf711e242ecbeb902f250f19fa68300"} Sep 09 23:54:11.987722 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:11.987701 2574 generic.go:356] "Generic (PLEG): container finished" podID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerID="0f8599c88e8cfc4b440b3fef63a5df51bd74e144ef12ddd6a0b3db4b17f0f636" exitCode=0 Sep 09 23:54:11.987809 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:11.987768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" event={"ID":"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb","Type":"ContainerDied","Data":"0f8599c88e8cfc4b440b3fef63a5df51bd74e144ef12ddd6a0b3db4b17f0f636"} Sep 09 23:54:12.994317 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:12.994263 2574 generic.go:356] "Generic (PLEG): container finished" podID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerID="b21afbd1c48d0ef05ddcc8b191c4c19a0a52e3e0aed6e77271105e539e270dd1" exitCode=0 Sep 09 23:54:12.994697 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:12.994351 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" event={"ID":"32b9103a-e59f-49ad-8b4b-80c42c68fea5","Type":"ContainerDied","Data":"b21afbd1c48d0ef05ddcc8b191c4c19a0a52e3e0aed6e77271105e539e270dd1"} Sep 09 23:54:12.996590 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:12.996569 2574 generic.go:356] "Generic (PLEG): container finished" podID="abad69f7-2b89-4c51-811d-fbd9be538345" containerID="11db45562d062384eaaa2f509e2caf01c65b4fee062f62a44e5a415daef02c25" exitCode=0 Sep 09 23:54:12.996678 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:12.996599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" event={"ID":"abad69f7-2b89-4c51-811d-fbd9be538345","Type":"ContainerDied","Data":"11db45562d062384eaaa2f509e2caf01c65b4fee062f62a44e5a415daef02c25"} Sep 09 23:54:12.998699 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:12.998677 2574 generic.go:356] "Generic (PLEG): container finished" podID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerID="58b096fab033a3ac2a16463dc93a25b2ff4f87075af65ec4de325c643302c5f8" exitCode=0 Sep 09 23:54:12.998787 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:12.998703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" event={"ID":"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb","Type":"ContainerDied","Data":"58b096fab033a3ac2a16463dc93a25b2ff4f87075af65ec4de325c643302c5f8"} Sep 09 23:54:14.005901 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.005866 2574 generic.go:356] "Generic (PLEG): container finished" podID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerID="d7429ebdbe07878ec47e24087d90633d6134a92c7450d26d2ba23b688f171a60" exitCode=0 Sep 09 23:54:14.006258 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.005943 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" event={"ID":"32b9103a-e59f-49ad-8b4b-80c42c68fea5","Type":"ContainerDied","Data":"d7429ebdbe07878ec47e24087d90633d6134a92c7450d26d2ba23b688f171a60"} Sep 09 23:54:14.125262 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.125232 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:14.162096 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.162073 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-bundle\") pod \"abad69f7-2b89-4c51-811d-fbd9be538345\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " Sep 09 23:54:14.162217 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.162116 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6ns9\" (UniqueName: \"kubernetes.io/projected/abad69f7-2b89-4c51-811d-fbd9be538345-kube-api-access-x6ns9\") pod \"abad69f7-2b89-4c51-811d-fbd9be538345\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " Sep 09 23:54:14.162217 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.162078 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:14.162217 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.162173 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-util\") pod \"abad69f7-2b89-4c51-811d-fbd9be538345\" (UID: \"abad69f7-2b89-4c51-811d-fbd9be538345\") " Sep 09 23:54:14.162884 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.162847 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-bundle" (OuterVolumeSpecName: "bundle") pod "abad69f7-2b89-4c51-811d-fbd9be538345" (UID: "abad69f7-2b89-4c51-811d-fbd9be538345"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:54:14.164190 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.164165 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abad69f7-2b89-4c51-811d-fbd9be538345-kube-api-access-x6ns9" (OuterVolumeSpecName: "kube-api-access-x6ns9") pod "abad69f7-2b89-4c51-811d-fbd9be538345" (UID: "abad69f7-2b89-4c51-811d-fbd9be538345"). InnerVolumeSpecName "kube-api-access-x6ns9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:54:14.169256 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.169232 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-util" (OuterVolumeSpecName: "util") pod "abad69f7-2b89-4c51-811d-fbd9be538345" (UID: "abad69f7-2b89-4c51-811d-fbd9be538345"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:54:14.263372 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.263317 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-util\") pod \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " Sep 09 23:54:14.263372 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.263355 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htnv4\" (UniqueName: \"kubernetes.io/projected/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-kube-api-access-htnv4\") pod \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " Sep 09 23:54:14.263514 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.263400 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-bundle\") pod \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\" (UID: \"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb\") " Sep 09 23:54:14.263603 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.263587 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-util\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:14.263679 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.263609 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abad69f7-2b89-4c51-811d-fbd9be538345-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:14.263679 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.263625 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ns9\" (UniqueName: \"kubernetes.io/projected/abad69f7-2b89-4c51-811d-fbd9be538345-kube-api-access-x6ns9\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:14.263940 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.263905 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-bundle" (OuterVolumeSpecName: "bundle") pod "510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" (UID: "510c52cc-e02c-4d7b-a0e5-3cea4b6201eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:54:14.265462 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.265440 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-kube-api-access-htnv4" (OuterVolumeSpecName: "kube-api-access-htnv4") pod "510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" (UID: "510c52cc-e02c-4d7b-a0e5-3cea4b6201eb"). InnerVolumeSpecName "kube-api-access-htnv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:54:14.268696 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.268674 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-util" (OuterVolumeSpecName: "util") pod "510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" (UID: "510c52cc-e02c-4d7b-a0e5-3cea4b6201eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:54:14.369725 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.365506 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-util\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:14.369725 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.365558 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htnv4\" (UniqueName: \"kubernetes.io/projected/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-kube-api-access-htnv4\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:14.369725 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:14.365586 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510c52cc-e02c-4d7b-a0e5-3cea4b6201eb-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:15.012673 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.012639 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" Sep 09 23:54:15.013120 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.012641 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7drk" event={"ID":"abad69f7-2b89-4c51-811d-fbd9be538345","Type":"ContainerDied","Data":"090590bdcddca182293d0ac10e3ded2925d22c2e56c30ec42d0cc63b55de4f74"} Sep 09 23:54:15.013120 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.012804 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090590bdcddca182293d0ac10e3ded2925d22c2e56c30ec42d0cc63b55de4f74" Sep 09 23:54:15.014808 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.014778 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" event={"ID":"510c52cc-e02c-4d7b-a0e5-3cea4b6201eb","Type":"ContainerDied","Data":"abc4c175240370ef3852a2a0aa56982e5a64b7fe6f1a2469b7ef9ece37232d7c"} Sep 09 23:54:15.014910 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.014825 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abc4c175240370ef3852a2a0aa56982e5a64b7fe6f1a2469b7ef9ece37232d7c" Sep 09 23:54:15.014998 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.014982 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6btt87" Sep 09 23:54:15.139334 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.139314 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:15.172323 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.172296 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxp8c\" (UniqueName: \"kubernetes.io/projected/32b9103a-e59f-49ad-8b4b-80c42c68fea5-kube-api-access-rxp8c\") pod \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " Sep 09 23:54:15.172468 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.172340 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-bundle\") pod \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " Sep 09 23:54:15.172468 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.172388 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-util\") pod \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\" (UID: \"32b9103a-e59f-49ad-8b4b-80c42c68fea5\") " Sep 09 23:54:15.172888 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.172862 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-bundle" (OuterVolumeSpecName: "bundle") pod "32b9103a-e59f-49ad-8b4b-80c42c68fea5" (UID: "32b9103a-e59f-49ad-8b4b-80c42c68fea5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:54:15.174378 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.174348 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b9103a-e59f-49ad-8b4b-80c42c68fea5-kube-api-access-rxp8c" (OuterVolumeSpecName: "kube-api-access-rxp8c") pod "32b9103a-e59f-49ad-8b4b-80c42c68fea5" (UID: "32b9103a-e59f-49ad-8b4b-80c42c68fea5"). InnerVolumeSpecName "kube-api-access-rxp8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:54:15.177962 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.177937 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-util" (OuterVolumeSpecName: "util") pod "32b9103a-e59f-49ad-8b4b-80c42c68fea5" (UID: "32b9103a-e59f-49ad-8b4b-80c42c68fea5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:54:15.273581 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.273511 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxp8c\" (UniqueName: \"kubernetes.io/projected/32b9103a-e59f-49ad-8b4b-80c42c68fea5-kube-api-access-rxp8c\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:15.273581 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.273538 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:15.273581 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:15.273551 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32b9103a-e59f-49ad-8b4b-80c42c68fea5-util\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:16.021578 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:16.021546 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" Sep 09 23:54:16.021578 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:16.021561 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ld2c" event={"ID":"32b9103a-e59f-49ad-8b4b-80c42c68fea5","Type":"ContainerDied","Data":"d7a2b4d444b44db0d5681bbd553d4f3fd3fb73c2fbbcd3bd96d6865676d8fa82"} Sep 09 23:54:16.022000 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:16.021605 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7a2b4d444b44db0d5681bbd553d4f3fd3fb73c2fbbcd3bd96d6865676d8fa82" Sep 09 23:54:23.312074 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312042 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dc545d574-2nlvs"] Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312340 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerName="pull" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312353 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerName="pull" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312365 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abad69f7-2b89-4c51-811d-fbd9be538345" containerName="util" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312370 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="abad69f7-2b89-4c51-811d-fbd9be538345" containerName="util" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312376 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerName="extract" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312382 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerName="extract" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312392 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abad69f7-2b89-4c51-811d-fbd9be538345" containerName="pull" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312397 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="abad69f7-2b89-4c51-811d-fbd9be538345" containerName="pull" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312403 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abad69f7-2b89-4c51-811d-fbd9be538345" containerName="extract" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312408 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="abad69f7-2b89-4c51-811d-fbd9be538345" containerName="extract" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312414 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerName="util" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312419 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerName="util" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312424 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerName="pull" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312429 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerName="pull" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312437 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerName="util" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312442 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerName="util" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312455 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerName="extract" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312460 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerName="extract" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312504 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="510c52cc-e02c-4d7b-a0e5-3cea4b6201eb" containerName="extract" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312513 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="abad69f7-2b89-4c51-811d-fbd9be538345" containerName="extract" Sep 09 23:54:23.312550 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.312521 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b9103a-e59f-49ad-8b4b-80c42c68fea5" containerName="extract" Sep 09 23:54:23.317476 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.317460 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.326967 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.326941 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc545d574-2nlvs"] Sep 09 23:54:23.432145 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.432119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5866a017-4f1f-4e48-afb8-7475194eb0fc-console-serving-cert\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.432311 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.432158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-oauth-serving-cert\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.432311 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.432178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5866a017-4f1f-4e48-afb8-7475194eb0fc-console-oauth-config\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.432311 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.432197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-service-ca\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.432311 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.432219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-trusted-ca-bundle\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.432311 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.432243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-console-config\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.432311 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.432258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5bw\" (UniqueName: \"kubernetes.io/projected/5866a017-4f1f-4e48-afb8-7475194eb0fc-kube-api-access-xs5bw\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.533526 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.533502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5866a017-4f1f-4e48-afb8-7475194eb0fc-console-serving-cert\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.533650 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.533538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-oauth-serving-cert\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.533650 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.533558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5866a017-4f1f-4e48-afb8-7475194eb0fc-console-oauth-config\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.533650 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.533577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-service-ca\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.533826 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.533674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-trusted-ca-bundle\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.533826 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.533712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-console-config\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.533826 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.533733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5bw\" (UniqueName: \"kubernetes.io/projected/5866a017-4f1f-4e48-afb8-7475194eb0fc-kube-api-access-xs5bw\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.534434 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.534413 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-console-config\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.534434 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.534423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-service-ca\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.534572 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.534415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-oauth-serving-cert\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.534656 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.534634 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5866a017-4f1f-4e48-afb8-7475194eb0fc-trusted-ca-bundle\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.535924 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.535907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5866a017-4f1f-4e48-afb8-7475194eb0fc-console-oauth-config\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.536180 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.536163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5866a017-4f1f-4e48-afb8-7475194eb0fc-console-serving-cert\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.541452 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.541430 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5bw\" (UniqueName: \"kubernetes.io/projected/5866a017-4f1f-4e48-afb8-7475194eb0fc-kube-api-access-xs5bw\") pod \"console-6dc545d574-2nlvs\" (UID: \"5866a017-4f1f-4e48-afb8-7475194eb0fc\") " pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.627101 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.627050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:23.896914 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:23.896891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc545d574-2nlvs"] Sep 09 23:54:23.898715 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:54:23.898680 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5866a017_4f1f_4e48_afb8_7475194eb0fc.slice/crio-e2f7633af9a4749a9c43fbda3405ab5b33e8a973031cc091293c52e3e9ea02e1 WatchSource:0}: Error finding container e2f7633af9a4749a9c43fbda3405ab5b33e8a973031cc091293c52e3e9ea02e1: Status 404 returned error can't find the container with id e2f7633af9a4749a9c43fbda3405ab5b33e8a973031cc091293c52e3e9ea02e1 Sep 09 23:54:24.062447 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:24.062414 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc545d574-2nlvs" event={"ID":"5866a017-4f1f-4e48-afb8-7475194eb0fc","Type":"ContainerStarted","Data":"928d958f2e7db89cdda4f7ca8c598658937d9554e19b5c3052479c972f0179c8"} Sep 09 23:54:24.062607 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:24.062470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc545d574-2nlvs" event={"ID":"5866a017-4f1f-4e48-afb8-7475194eb0fc","Type":"ContainerStarted","Data":"e2f7633af9a4749a9c43fbda3405ab5b33e8a973031cc091293c52e3e9ea02e1"} Sep 09 23:54:24.085955 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:24.085905 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dc545d574-2nlvs" podStartSLOduration=1.085892233 podStartE2EDuration="1.085892233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:54:24.078136113 +0000 UTC m=+376.285275883" watchObservedRunningTime="2026-09-09 23:54:24.085892233 +0000 UTC m=+376.293032012" Sep 09 23:54:33.627255 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:33.627220 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:33.627645 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:33.627312 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:33.632039 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:33.632018 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:34.112674 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:34.112645 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dc545d574-2nlvs" Sep 09 23:54:34.158037 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:34.158007 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575fd5647d-9t52x"] Sep 09 23:54:40.139249 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:40.139211 2574 generic.go:356] "Generic (PLEG): container finished" podID="b37a473a-3451-4ce8-bee6-9576a593f605" containerID="f6bd96fceb870dc7d9fd30de3f23a755d4247fd41479b993c97b3328b681a5b3" exitCode=0 Sep 09 23:54:40.139640 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:40.139320 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" event={"ID":"b37a473a-3451-4ce8-bee6-9576a593f605","Type":"ContainerDied","Data":"f6bd96fceb870dc7d9fd30de3f23a755d4247fd41479b993c97b3328b681a5b3"} Sep 09 23:54:41.146157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:41.146124 2574 generic.go:356] "Generic (PLEG): container finished" podID="b37a473a-3451-4ce8-bee6-9576a593f605" containerID="df7c7330f7aed8d6407689ac7177dbef98fd171936f4975a84752c3fb2fb97bf" exitCode=0 Sep 09 23:54:41.146541 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:41.146213 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" event={"ID":"b37a473a-3451-4ce8-bee6-9576a593f605","Type":"ContainerDied","Data":"df7c7330f7aed8d6407689ac7177dbef98fd171936f4975a84752c3fb2fb97bf"} Sep 09 23:54:42.276363 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.276341 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:42.379975 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.379947 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-util\") pod \"b37a473a-3451-4ce8-bee6-9576a593f605\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " Sep 09 23:54:42.380097 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.380003 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7crrx\" (UniqueName: \"kubernetes.io/projected/b37a473a-3451-4ce8-bee6-9576a593f605-kube-api-access-7crrx\") pod \"b37a473a-3451-4ce8-bee6-9576a593f605\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " Sep 09 23:54:42.380097 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.380041 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-bundle\") pod \"b37a473a-3451-4ce8-bee6-9576a593f605\" (UID: \"b37a473a-3451-4ce8-bee6-9576a593f605\") " Sep 09 23:54:42.380527 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.380506 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-bundle" (OuterVolumeSpecName: "bundle") pod "b37a473a-3451-4ce8-bee6-9576a593f605" (UID: "b37a473a-3451-4ce8-bee6-9576a593f605"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:54:42.382051 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.382022 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37a473a-3451-4ce8-bee6-9576a593f605-kube-api-access-7crrx" (OuterVolumeSpecName: "kube-api-access-7crrx") pod "b37a473a-3451-4ce8-bee6-9576a593f605" (UID: "b37a473a-3451-4ce8-bee6-9576a593f605"). InnerVolumeSpecName "kube-api-access-7crrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:54:42.384412 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.384391 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-util" (OuterVolumeSpecName: "util") pod "b37a473a-3451-4ce8-bee6-9576a593f605" (UID: "b37a473a-3451-4ce8-bee6-9576a593f605"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:54:42.481195 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.481129 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-util\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:42.481195 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.481154 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7crrx\" (UniqueName: \"kubernetes.io/projected/b37a473a-3451-4ce8-bee6-9576a593f605-kube-api-access-7crrx\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:42.481195 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:42.481163 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b37a473a-3451-4ce8-bee6-9576a593f605-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:43.158455 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:43.158428 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" Sep 09 23:54:43.158455 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:43.158450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rx7n6" event={"ID":"b37a473a-3451-4ce8-bee6-9576a593f605","Type":"ContainerDied","Data":"81964b672f007dc9f0f03c2f8c7c6a662860acdc8d8316fc5eaf9c81992f5473"} Sep 09 23:54:43.158653 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:43.158489 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81964b672f007dc9f0f03c2f8c7c6a662860acdc8d8316fc5eaf9c81992f5473" Sep 09 23:54:50.813830 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.813796 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-97plq"] Sep 09 23:54:50.814183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.814073 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37a473a-3451-4ce8-bee6-9576a593f605" containerName="util" Sep 09 23:54:50.814183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.814083 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37a473a-3451-4ce8-bee6-9576a593f605" containerName="util" Sep 09 23:54:50.814183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.814092 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37a473a-3451-4ce8-bee6-9576a593f605" containerName="extract" Sep 09 23:54:50.814183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.814098 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37a473a-3451-4ce8-bee6-9576a593f605" containerName="extract" Sep 09 23:54:50.814183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.814113 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37a473a-3451-4ce8-bee6-9576a593f605" containerName="pull" Sep 09 23:54:50.814183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.814119 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37a473a-3451-4ce8-bee6-9576a593f605" containerName="pull" Sep 09 23:54:50.814183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.814177 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37a473a-3451-4ce8-bee6-9576a593f605" containerName="extract" Sep 09 23:54:50.818468 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.818451 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-97plq" Sep 09 23:54:50.821453 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.821425 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wds7p\"" Sep 09 23:54:50.822250 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.822234 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 23:54:50.823243 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.823227 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 23:54:50.833404 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.833384 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-97plq"] Sep 09 23:54:50.942042 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:50.942022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5hdd\" (UniqueName: \"kubernetes.io/projected/54cbf93f-8a00-4ebd-9e63-61215a8ddfde-kube-api-access-m5hdd\") pod \"authorino-operator-76b478bd78-97plq\" (UID: \"54cbf93f-8a00-4ebd-9e63-61215a8ddfde\") " pod="kuadrant-system/authorino-operator-76b478bd78-97plq" Sep 09 23:54:51.042866 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:51.042843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5hdd\" (UniqueName: \"kubernetes.io/projected/54cbf93f-8a00-4ebd-9e63-61215a8ddfde-kube-api-access-m5hdd\") pod \"authorino-operator-76b478bd78-97plq\" (UID: \"54cbf93f-8a00-4ebd-9e63-61215a8ddfde\") " pod="kuadrant-system/authorino-operator-76b478bd78-97plq" Sep 09 23:54:51.057323 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:51.057297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5hdd\" (UniqueName: \"kubernetes.io/projected/54cbf93f-8a00-4ebd-9e63-61215a8ddfde-kube-api-access-m5hdd\") pod \"authorino-operator-76b478bd78-97plq\" (UID: \"54cbf93f-8a00-4ebd-9e63-61215a8ddfde\") " pod="kuadrant-system/authorino-operator-76b478bd78-97plq" Sep 09 23:54:51.128578 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:51.128504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-97plq" Sep 09 23:54:51.403397 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:51.403373 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-97plq"] Sep 09 23:54:51.405078 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:54:51.405054 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cbf93f_8a00_4ebd_9e63_61215a8ddfde.slice/crio-4809020faa705477f98cc101d3571727df4e20899d80a7b7fbf6f0ef46e66c21 WatchSource:0}: Error finding container 4809020faa705477f98cc101d3571727df4e20899d80a7b7fbf6f0ef46e66c21: Status 404 returned error can't find the container with id 4809020faa705477f98cc101d3571727df4e20899d80a7b7fbf6f0ef46e66c21 Sep 09 23:54:52.204359 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:52.204329 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-97plq" event={"ID":"54cbf93f-8a00-4ebd-9e63-61215a8ddfde","Type":"ContainerStarted","Data":"4809020faa705477f98cc101d3571727df4e20899d80a7b7fbf6f0ef46e66c21"} Sep 09 23:54:54.217106 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:54.217075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-97plq" event={"ID":"54cbf93f-8a00-4ebd-9e63-61215a8ddfde","Type":"ContainerStarted","Data":"573080132eda0049fee9d219e217e5c6d3c614e5d8b5be8ca3a33b54ec25887b"} Sep 09 23:54:54.217484 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:54.217232 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-97plq" Sep 09 23:54:54.235755 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:54.235701 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-97plq" podStartSLOduration=1.565174978 podStartE2EDuration="4.2356816s" totalImagesPullingTime="2.670506622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:54:54.232610776 +0000 UTC m=+406.439750546" watchObservedRunningTime="2026-09-09 23:54:54.2356816 +0000 UTC m=+406.442821370" Sep 09 23:54:54.938947 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:54.938909 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7"] Sep 09 23:54:54.942524 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:54.942506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:54:54.945010 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:54.944990 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bqn76\"" Sep 09 23:54:54.954255 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:54.954233 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7"] Sep 09 23:54:55.079070 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:55.079043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr548\" (UniqueName: \"kubernetes.io/projected/ee982f0b-5f28-4e66-99cb-bd30bd44193a-kube-api-access-lr548\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" (UID: \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:54:55.079201 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:55.079075 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee982f0b-5f28-4e66-99cb-bd30bd44193a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" (UID: \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:54:55.179531 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:55.179504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr548\" (UniqueName: \"kubernetes.io/projected/ee982f0b-5f28-4e66-99cb-bd30bd44193a-kube-api-access-lr548\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" (UID: \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:54:55.179644 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:55.179535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee982f0b-5f28-4e66-99cb-bd30bd44193a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" (UID: \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:54:55.179844 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:55.179829 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee982f0b-5f28-4e66-99cb-bd30bd44193a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" (UID: \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:54:55.191611 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:55.191555 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr548\" (UniqueName: \"kubernetes.io/projected/ee982f0b-5f28-4e66-99cb-bd30bd44193a-kube-api-access-lr548\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" (UID: \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:54:55.251624 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:55.251596 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:54:55.532873 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:55.532847 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7"] Sep 09 23:54:55.533846 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:54:55.533810 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee982f0b_5f28_4e66_99cb_bd30bd44193a.slice/crio-0488afabd359ad7053678749a98312452f165d80b13ce0055e2f59f88a26bda1 WatchSource:0}: Error finding container 0488afabd359ad7053678749a98312452f165d80b13ce0055e2f59f88a26bda1: Status 404 returned error can't find the container with id 0488afabd359ad7053678749a98312452f165d80b13ce0055e2f59f88a26bda1 Sep 09 23:54:56.229811 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:56.229772 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" event={"ID":"ee982f0b-5f28-4e66-99cb-bd30bd44193a","Type":"ContainerStarted","Data":"0488afabd359ad7053678749a98312452f165d80b13ce0055e2f59f88a26bda1"} Sep 09 23:54:57.101475 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.101443 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq"] Sep 09 23:54:57.105975 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.105959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" Sep 09 23:54:57.108331 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.108302 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 09 23:54:57.108450 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.108367 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qgkzn\"" Sep 09 23:54:57.118490 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.118469 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq"] Sep 09 23:54:57.194762 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.194737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74jfb\" (UniqueName: \"kubernetes.io/projected/72e5a4e2-a0d3-4521-a090-d28f7d649dc4-kube-api-access-74jfb\") pod \"dns-operator-controller-manager-7b84cd7c9c-w58fq\" (UID: \"72e5a4e2-a0d3-4521-a090-d28f7d649dc4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" Sep 09 23:54:57.295549 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.295524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74jfb\" (UniqueName: \"kubernetes.io/projected/72e5a4e2-a0d3-4521-a090-d28f7d649dc4-kube-api-access-74jfb\") pod \"dns-operator-controller-manager-7b84cd7c9c-w58fq\" (UID: \"72e5a4e2-a0d3-4521-a090-d28f7d649dc4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" Sep 09 23:54:57.307092 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.307069 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74jfb\" (UniqueName: \"kubernetes.io/projected/72e5a4e2-a0d3-4521-a090-d28f7d649dc4-kube-api-access-74jfb\") pod \"dns-operator-controller-manager-7b84cd7c9c-w58fq\" (UID: \"72e5a4e2-a0d3-4521-a090-d28f7d649dc4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" Sep 09 23:54:57.417095 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.417014 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" Sep 09 23:54:57.691022 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:57.690990 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq"] Sep 09 23:54:57.692440 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:54:57.692417 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e5a4e2_a0d3_4521_a090_d28f7d649dc4.slice/crio-8edf8ad0568c4e54ea3d3c9d2142e4da1ffa069a49993751993505b2bcd0a3e8 WatchSource:0}: Error finding container 8edf8ad0568c4e54ea3d3c9d2142e4da1ffa069a49993751993505b2bcd0a3e8: Status 404 returned error can't find the container with id 8edf8ad0568c4e54ea3d3c9d2142e4da1ffa069a49993751993505b2bcd0a3e8 Sep 09 23:54:58.244482 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:58.244449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" event={"ID":"72e5a4e2-a0d3-4521-a090-d28f7d649dc4","Type":"ContainerStarted","Data":"8edf8ad0568c4e54ea3d3c9d2142e4da1ffa069a49993751993505b2bcd0a3e8"} Sep 09 23:54:59.184603 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.184511 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-575fd5647d-9t52x" podUID="f9c58547-94e1-4618-9b52-d48b9d267427" containerName="console" containerID="cri-o://0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e" gracePeriod=15 Sep 09 23:54:59.385494 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.385437 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28"] Sep 09 23:54:59.389387 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.389369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:54:59.394214 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.394192 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-m9kdk\"" Sep 09 23:54:59.399621 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.399597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28"] Sep 09 23:54:59.502594 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.502571 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:54:59.512662 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.512466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnp27\" (UniqueName: \"kubernetes.io/projected/18a39a25-328d-4392-9c69-1d9dacb89fb1-kube-api-access-dnp27\") pod \"limitador-operator-controller-manager-74495c69f8-kzm28\" (UID: \"18a39a25-328d-4392-9c69-1d9dacb89fb1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:54:59.613205 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.613174 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml9vs\" (UniqueName: \"kubernetes.io/projected/f9c58547-94e1-4618-9b52-d48b9d267427-kube-api-access-ml9vs\") pod \"f9c58547-94e1-4618-9b52-d48b9d267427\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " Sep 09 23:54:59.613405 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.613225 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-oauth-config\") pod \"f9c58547-94e1-4618-9b52-d48b9d267427\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " Sep 09 23:54:59.613405 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.613270 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-trusted-ca-bundle\") pod \"f9c58547-94e1-4618-9b52-d48b9d267427\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " Sep 09 23:54:59.613405 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.613342 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-console-config\") pod \"f9c58547-94e1-4618-9b52-d48b9d267427\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " Sep 09 23:54:59.613405 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.613392 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-service-ca\") pod \"f9c58547-94e1-4618-9b52-d48b9d267427\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " Sep 09 23:54:59.613698 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.613430 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-serving-cert\") pod \"f9c58547-94e1-4618-9b52-d48b9d267427\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " Sep 09 23:54:59.613698 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.613462 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-oauth-serving-cert\") pod \"f9c58547-94e1-4618-9b52-d48b9d267427\" (UID: \"f9c58547-94e1-4618-9b52-d48b9d267427\") " Sep 09 23:54:59.613698 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.613666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnp27\" (UniqueName: \"kubernetes.io/projected/18a39a25-328d-4392-9c69-1d9dacb89fb1-kube-api-access-dnp27\") pod \"limitador-operator-controller-manager-74495c69f8-kzm28\" (UID: \"18a39a25-328d-4392-9c69-1d9dacb89fb1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:54:59.613960 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.613918 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-service-ca" (OuterVolumeSpecName: "service-ca") pod "f9c58547-94e1-4618-9b52-d48b9d267427" (UID: "f9c58547-94e1-4618-9b52-d48b9d267427"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:54:59.614182 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.614150 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f9c58547-94e1-4618-9b52-d48b9d267427" (UID: "f9c58547-94e1-4618-9b52-d48b9d267427"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:54:59.614316 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.614189 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-console-config" (OuterVolumeSpecName: "console-config") pod "f9c58547-94e1-4618-9b52-d48b9d267427" (UID: "f9c58547-94e1-4618-9b52-d48b9d267427"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:54:59.614388 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.614265 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f9c58547-94e1-4618-9b52-d48b9d267427" (UID: "f9c58547-94e1-4618-9b52-d48b9d267427"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:54:59.615844 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.615814 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c58547-94e1-4618-9b52-d48b9d267427-kube-api-access-ml9vs" (OuterVolumeSpecName: "kube-api-access-ml9vs") pod "f9c58547-94e1-4618-9b52-d48b9d267427" (UID: "f9c58547-94e1-4618-9b52-d48b9d267427"). InnerVolumeSpecName "kube-api-access-ml9vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:54:59.615980 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.615955 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f9c58547-94e1-4618-9b52-d48b9d267427" (UID: "f9c58547-94e1-4618-9b52-d48b9d267427"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:54:59.616300 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.616260 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f9c58547-94e1-4618-9b52-d48b9d267427" (UID: "f9c58547-94e1-4618-9b52-d48b9d267427"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:54:59.624108 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.624083 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnp27\" (UniqueName: \"kubernetes.io/projected/18a39a25-328d-4392-9c69-1d9dacb89fb1-kube-api-access-dnp27\") pod \"limitador-operator-controller-manager-74495c69f8-kzm28\" (UID: \"18a39a25-328d-4392-9c69-1d9dacb89fb1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:54:59.714221 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.714190 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-trusted-ca-bundle\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:59.714221 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.714224 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-console-config\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:59.714465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.714237 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-service-ca\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:59.714465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.714262 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-serving-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:59.714465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.714274 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9c58547-94e1-4618-9b52-d48b9d267427-oauth-serving-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:59.714465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.714301 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml9vs\" (UniqueName: \"kubernetes.io/projected/f9c58547-94e1-4618-9b52-d48b9d267427-kube-api-access-ml9vs\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:59.714465 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.714310 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9c58547-94e1-4618-9b52-d48b9d267427-console-oauth-config\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:54:59.798153 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:54:59.798124 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:00.088021 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.087628 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28"] Sep 09 23:55:00.258827 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.258769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" event={"ID":"18a39a25-328d-4392-9c69-1d9dacb89fb1","Type":"ContainerStarted","Data":"0db8e0e4057770f7cb2359b991d20b3e3cba3d54791584c07d3c8ee9f8cd4ccd"} Sep 09 23:55:00.260153 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.260123 2574 generic.go:356] "Generic (PLEG): container finished" podID="f9c58547-94e1-4618-9b52-d48b9d267427" containerID="0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e" exitCode=0 Sep 09 23:55:00.260254 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.260181 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575fd5647d-9t52x" Sep 09 23:55:00.260254 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.260201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575fd5647d-9t52x" event={"ID":"f9c58547-94e1-4618-9b52-d48b9d267427","Type":"ContainerDied","Data":"0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e"} Sep 09 23:55:00.260254 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.260244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575fd5647d-9t52x" event={"ID":"f9c58547-94e1-4618-9b52-d48b9d267427","Type":"ContainerDied","Data":"17bd5e20d5140b499655aef038b16aed3565d4f0e2084bb13d7e61a19fcc0bf5"} Sep 09 23:55:00.260403 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.260263 2574 scope.go:117] "RemoveContainer" containerID="0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e" Sep 09 23:55:00.262430 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.262098 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" event={"ID":"72e5a4e2-a0d3-4521-a090-d28f7d649dc4","Type":"ContainerStarted","Data":"314d20a8deb4f27f1f83de84bed46e792456c5ea6a3feb65b3c08729b317c73a"} Sep 09 23:55:00.262430 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.262223 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" Sep 09 23:55:00.269227 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.269211 2574 scope.go:117] "RemoveContainer" containerID="0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e" Sep 09 23:55:00.269520 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:00.269499 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e\": container with ID starting with 0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e not found: ID does not exist" containerID="0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e" Sep 09 23:55:00.269558 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.269526 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e"} err="failed to get container status \"0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e\": rpc error: code = NotFound desc = could not find container \"0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e\": container with ID starting with 0c8818be190f00624ccf171e837d04dc24f79cebbc1462da3b283968310d112e not found: ID does not exist" Sep 09 23:55:00.282764 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.282725 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" podStartSLOduration=1.606694249 podStartE2EDuration="3.282713367s" totalImagesPullingTime="1.676019118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:55:00.281781437 +0000 UTC m=+412.488921206" watchObservedRunningTime="2026-09-09 23:55:00.282713367 +0000 UTC m=+412.489853137" Sep 09 23:55:00.296507 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.296483 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575fd5647d-9t52x"] Sep 09 23:55:00.306301 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:00.306250 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-575fd5647d-9t52x"] Sep 09 23:55:02.240126 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:02.240044 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c58547-94e1-4618-9b52-d48b9d267427" path="/var/lib/kubelet/pods/f9c58547-94e1-4618-9b52-d48b9d267427/volumes" Sep 09 23:55:02.277300 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:02.277256 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" event={"ID":"ee982f0b-5f28-4e66-99cb-bd30bd44193a","Type":"ContainerStarted","Data":"0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4"} Sep 09 23:55:02.277469 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:02.277405 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:02.298447 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:02.298394 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" podStartSLOduration=2.026289735 podStartE2EDuration="8.298381454s" totalImagesPullingTime="6.272091719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:55:02.295845711 +0000 UTC m=+414.502985521" watchObservedRunningTime="2026-09-09 23:55:02.298381454 +0000 UTC m=+414.505521236" Sep 09 23:55:03.283980 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:03.283948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" event={"ID":"18a39a25-328d-4392-9c69-1d9dacb89fb1","Type":"ContainerStarted","Data":"c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1"} Sep 09 23:55:03.284482 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:03.284178 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:03.299694 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:03.299647 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" podStartSLOduration=1.732319941 podStartE2EDuration="4.299631738s" totalImagesPullingTime="2.567311797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:55:03.299082917 +0000 UTC m=+415.506222686" watchObservedRunningTime="2026-09-09 23:55:03.299631738 +0000 UTC m=+415.506771511" Sep 09 23:55:04.463004 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.462970 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c"] Sep 09 23:55:04.466632 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.464173 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c58547-94e1-4618-9b52-d48b9d267427" containerName="console" Sep 09 23:55:04.466632 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.464207 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c58547-94e1-4618-9b52-d48b9d267427" containerName="console" Sep 09 23:55:04.466632 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.464404 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c58547-94e1-4618-9b52-d48b9d267427" containerName="console" Sep 09 23:55:04.469060 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.469033 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.471698 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.471672 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 09 23:55:04.472141 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.471969 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 09 23:55:04.472659 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.472641 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lqn7r\"" Sep 09 23:55:04.474538 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.474515 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c"] Sep 09 23:55:04.659625 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.659593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e75f2bc6-cc62-4eed-86ee-4baad44a3a8f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mc97c\" (UID: \"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.659778 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.659638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e75f2bc6-cc62-4eed-86ee-4baad44a3a8f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mc97c\" (UID: \"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.659778 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.659683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/e75f2bc6-cc62-4eed-86ee-4baad44a3a8f-kube-api-access-zqhh9\") pod \"kuadrant-console-plugin-fbdd4f445-mc97c\" (UID: \"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.760982 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.760916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e75f2bc6-cc62-4eed-86ee-4baad44a3a8f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mc97c\" (UID: \"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.760982 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.760965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/e75f2bc6-cc62-4eed-86ee-4baad44a3a8f-kube-api-access-zqhh9\") pod \"kuadrant-console-plugin-fbdd4f445-mc97c\" (UID: \"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.761160 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.761136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e75f2bc6-cc62-4eed-86ee-4baad44a3a8f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mc97c\" (UID: \"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.761734 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.761710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e75f2bc6-cc62-4eed-86ee-4baad44a3a8f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mc97c\" (UID: \"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.763310 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.763277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e75f2bc6-cc62-4eed-86ee-4baad44a3a8f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mc97c\" (UID: \"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.768044 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.768021 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/e75f2bc6-cc62-4eed-86ee-4baad44a3a8f-kube-api-access-zqhh9\") pod \"kuadrant-console-plugin-fbdd4f445-mc97c\" (UID: \"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:04.779963 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:04.779945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" Sep 09 23:55:05.047663 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:05.047634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c"] Sep 09 23:55:05.049303 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:55:05.049266 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode75f2bc6_cc62_4eed_86ee_4baad44a3a8f.slice/crio-97d7366954d4127fdc5b1aee38b2f19b2970df1bda83c597aa8ddb8d2c4a4482 WatchSource:0}: Error finding container 97d7366954d4127fdc5b1aee38b2f19b2970df1bda83c597aa8ddb8d2c4a4482: Status 404 returned error can't find the container with id 97d7366954d4127fdc5b1aee38b2f19b2970df1bda83c597aa8ddb8d2c4a4482 Sep 09 23:55:05.223909 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:05.223884 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-97plq" Sep 09 23:55:05.296030 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:05.295996 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" event={"ID":"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f","Type":"ContainerStarted","Data":"97d7366954d4127fdc5b1aee38b2f19b2970df1bda83c597aa8ddb8d2c4a4482"} Sep 09 23:55:10.330397 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:10.330353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" event={"ID":"e75f2bc6-cc62-4eed-86ee-4baad44a3a8f","Type":"ContainerStarted","Data":"85741f1cd8ec240b5c9f5550c8a71a91250c340454f6e0bf8c96033349742e6c"} Sep 09 23:55:10.347268 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:10.347203 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mc97c" podStartSLOduration=1.915654382 podStartE2EDuration="6.34718897s" totalImagesPullingTime="4.431534588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:55:10.346086012 +0000 UTC m=+422.553225799" watchObservedRunningTime="2026-09-09 23:55:10.34718897 +0000 UTC m=+422.554328739" Sep 09 23:55:11.272613 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:11.272584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w58fq" Sep 09 23:55:13.285969 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:13.285940 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:14.291774 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:14.291746 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:15.259073 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.259038 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7"] Sep 09 23:55:15.259246 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.259222 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" containerName="manager" containerID="cri-o://0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4" gracePeriod=2 Sep 09 23:55:15.278211 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.278186 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7"] Sep 09 23:55:15.279722 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.279684 2574 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ee982f0b-5f28-4e66-99cb-bd30bd44193a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-5mln7\": pods \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:15.286662 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.286626 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28"] Sep 09 23:55:15.286944 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.286897 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" containerName="manager" containerID="cri-o://c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1" gracePeriod=2 Sep 09 23:55:15.289176 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.289141 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kzm28\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:15.289724 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.289704 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn"] Sep 09 23:55:15.290180 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.290162 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" containerName="manager" Sep 09 23:55:15.290347 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.290332 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" containerName="manager" Sep 09 23:55:15.290564 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.290550 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" containerName="manager" Sep 09 23:55:15.291422 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.291393 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:15.294021 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.294002 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28"] Sep 09 23:55:15.294407 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.294172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:15.310654 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.310632 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn"] Sep 09 23:55:15.314930 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.314908 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt"] Sep 09 23:55:15.315380 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.315363 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" containerName="manager" Sep 09 23:55:15.315431 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.315384 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" containerName="manager" Sep 09 23:55:15.315497 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.315485 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" containerName="manager" Sep 09 23:55:15.318683 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.318665 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" Sep 09 23:55:15.330862 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.330842 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt"] Sep 09 23:55:15.335711 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.335679 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kzm28\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:15.337681 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.337653 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:15.339636 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.339612 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:15.344100 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.344069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mhj\" (UniqueName: \"kubernetes.io/projected/b763f4e5-4255-4227-b319-858a759cfc4d-kube-api-access-96mhj\") pod \"limitador-operator-controller-manager-74495c69f8-ncvpt\" (UID: \"b763f4e5-4255-4227-b319-858a759cfc4d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" Sep 09 23:55:15.362634 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.362604 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kzm28\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:15.445639 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.445600 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bzx\" (UniqueName: \"kubernetes.io/projected/48aefcbb-ecd8-4408-95e4-dc018e1481eb-kube-api-access-78bzx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mhrbn\" (UID: \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:15.445789 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.445648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mhj\" (UniqueName: \"kubernetes.io/projected/b763f4e5-4255-4227-b319-858a759cfc4d-kube-api-access-96mhj\") pod \"limitador-operator-controller-manager-74495c69f8-ncvpt\" (UID: \"b763f4e5-4255-4227-b319-858a759cfc4d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" Sep 09 23:55:15.445789 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.445676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48aefcbb-ecd8-4408-95e4-dc018e1481eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mhrbn\" (UID: \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:15.457074 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.457023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mhj\" (UniqueName: \"kubernetes.io/projected/b763f4e5-4255-4227-b319-858a759cfc4d-kube-api-access-96mhj\") pod \"limitador-operator-controller-manager-74495c69f8-ncvpt\" (UID: \"b763f4e5-4255-4227-b319-858a759cfc4d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" Sep 09 23:55:15.529113 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.529089 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:15.531072 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.531052 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:15.531362 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.531342 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kzm28\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:15.533259 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.533237 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:15.534976 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.534957 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kzm28\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:15.536761 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:15.536741 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5mln7\" is forbidden: User \"system:node:ip-10-0-129-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-102.ec2.internal' and this object" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:15.546031 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.546014 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnp27\" (UniqueName: \"kubernetes.io/projected/18a39a25-328d-4392-9c69-1d9dacb89fb1-kube-api-access-dnp27\") pod \"18a39a25-328d-4392-9c69-1d9dacb89fb1\" (UID: \"18a39a25-328d-4392-9c69-1d9dacb89fb1\") " Sep 09 23:55:15.546107 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.546051 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee982f0b-5f28-4e66-99cb-bd30bd44193a-extensions-socket-volume\") pod \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\" (UID: \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\") " Sep 09 23:55:15.546107 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.546076 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr548\" (UniqueName: \"kubernetes.io/projected/ee982f0b-5f28-4e66-99cb-bd30bd44193a-kube-api-access-lr548\") pod \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\" (UID: \"ee982f0b-5f28-4e66-99cb-bd30bd44193a\") " Sep 09 23:55:15.546214 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.546149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48aefcbb-ecd8-4408-95e4-dc018e1481eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mhrbn\" (UID: \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:15.546273 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.546213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bzx\" (UniqueName: \"kubernetes.io/projected/48aefcbb-ecd8-4408-95e4-dc018e1481eb-kube-api-access-78bzx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mhrbn\" (UID: \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:15.546461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.546429 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee982f0b-5f28-4e66-99cb-bd30bd44193a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ee982f0b-5f28-4e66-99cb-bd30bd44193a" (UID: "ee982f0b-5f28-4e66-99cb-bd30bd44193a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:55:15.546552 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.546542 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48aefcbb-ecd8-4408-95e4-dc018e1481eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mhrbn\" (UID: \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:15.548171 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.548140 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee982f0b-5f28-4e66-99cb-bd30bd44193a-kube-api-access-lr548" (OuterVolumeSpecName: "kube-api-access-lr548") pod "ee982f0b-5f28-4e66-99cb-bd30bd44193a" (UID: "ee982f0b-5f28-4e66-99cb-bd30bd44193a"). InnerVolumeSpecName "kube-api-access-lr548". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:55:15.548232 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.548188 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a39a25-328d-4392-9c69-1d9dacb89fb1-kube-api-access-dnp27" (OuterVolumeSpecName: "kube-api-access-dnp27") pod "18a39a25-328d-4392-9c69-1d9dacb89fb1" (UID: "18a39a25-328d-4392-9c69-1d9dacb89fb1"). InnerVolumeSpecName "kube-api-access-dnp27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:55:15.559348 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.559327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bzx\" (UniqueName: \"kubernetes.io/projected/48aefcbb-ecd8-4408-95e4-dc018e1481eb-kube-api-access-78bzx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mhrbn\" (UID: \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:15.647374 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.647349 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnp27\" (UniqueName: \"kubernetes.io/projected/18a39a25-328d-4392-9c69-1d9dacb89fb1-kube-api-access-dnp27\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:55:15.647374 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.647370 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee982f0b-5f28-4e66-99cb-bd30bd44193a-extensions-socket-volume\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:55:15.647500 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.647379 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr548\" (UniqueName: \"kubernetes.io/projected/ee982f0b-5f28-4e66-99cb-bd30bd44193a-kube-api-access-lr548\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:55:15.706390 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.706367 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:15.713373 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.713357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" Sep 09 23:55:15.992184 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.992152 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn"] Sep 09 23:55:15.992855 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:55:15.992824 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb763f4e5_4255_4227_b319_858a759cfc4d.slice/crio-2865407fb43ff47888a6507560130120f0dfee67d8fdc72b82b448839b88bd5b WatchSource:0}: Error finding container 2865407fb43ff47888a6507560130120f0dfee67d8fdc72b82b448839b88bd5b: Status 404 returned error can't find the container with id 2865407fb43ff47888a6507560130120f0dfee67d8fdc72b82b448839b88bd5b Sep 09 23:55:15.993866 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:55:15.993842 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48aefcbb_ecd8_4408_95e4_dc018e1481eb.slice/crio-2bd4b8a1dbdcc73c85f869c0a93e5ceaa13d564dc49ae0359626cef0ac00281a WatchSource:0}: Error finding container 2bd4b8a1dbdcc73c85f869c0a93e5ceaa13d564dc49ae0359626cef0ac00281a: Status 404 returned error can't find the container with id 2bd4b8a1dbdcc73c85f869c0a93e5ceaa13d564dc49ae0359626cef0ac00281a Sep 09 23:55:15.995519 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:15.995455 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt"] Sep 09 23:55:16.242115 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.242076 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a39a25-328d-4392-9c69-1d9dacb89fb1" path="/var/lib/kubelet/pods/18a39a25-328d-4392-9c69-1d9dacb89fb1/volumes" Sep 09 23:55:16.242484 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.242461 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" path="/var/lib/kubelet/pods/ee982f0b-5f28-4e66-99cb-bd30bd44193a/volumes" Sep 09 23:55:16.363790 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.363756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" event={"ID":"48aefcbb-ecd8-4408-95e4-dc018e1481eb","Type":"ContainerStarted","Data":"5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4"} Sep 09 23:55:16.364193 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.363809 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" event={"ID":"48aefcbb-ecd8-4408-95e4-dc018e1481eb","Type":"ContainerStarted","Data":"2bd4b8a1dbdcc73c85f869c0a93e5ceaa13d564dc49ae0359626cef0ac00281a"} Sep 09 23:55:16.364193 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.363857 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:16.365600 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.365574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" event={"ID":"b763f4e5-4255-4227-b319-858a759cfc4d","Type":"ContainerStarted","Data":"6b7f2d71a3bc6ec93d768fe9b0dc719b3b4a8c5643c6158ea78365c699f781d6"} Sep 09 23:55:16.365722 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.365626 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" Sep 09 23:55:16.365722 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.365652 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" event={"ID":"b763f4e5-4255-4227-b319-858a759cfc4d","Type":"ContainerStarted","Data":"2865407fb43ff47888a6507560130120f0dfee67d8fdc72b82b448839b88bd5b"} Sep 09 23:55:16.366929 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.366907 2574 generic.go:356] "Generic (PLEG): container finished" podID="ee982f0b-5f28-4e66-99cb-bd30bd44193a" containerID="0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4" exitCode=0 Sep 09 23:55:16.367030 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.366953 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5mln7" Sep 09 23:55:16.367030 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.366946 2574 scope.go:117] "RemoveContainer" containerID="0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4" Sep 09 23:55:16.368543 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.368523 2574 generic.go:356] "Generic (PLEG): container finished" podID="18a39a25-328d-4392-9c69-1d9dacb89fb1" containerID="c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1" exitCode=0 Sep 09 23:55:16.368621 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.368554 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kzm28" Sep 09 23:55:16.375571 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.375556 2574 scope.go:117] "RemoveContainer" containerID="0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4" Sep 09 23:55:16.375830 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:16.375812 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4\": container with ID starting with 0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4 not found: ID does not exist" containerID="0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4" Sep 09 23:55:16.375870 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.375838 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4"} err="failed to get container status \"0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4\": rpc error: code = NotFound desc = could not find container \"0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4\": container with ID starting with 0af5575f17a45235f76cf8f00402f9096d397d6b160f5e8f56acbcfd52723eb4 not found: ID does not exist" Sep 09 23:55:16.375870 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.375855 2574 scope.go:117] "RemoveContainer" containerID="c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1" Sep 09 23:55:16.381490 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.381452 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" podStartSLOduration=1.381440346 podStartE2EDuration="1.381440346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:55:16.380607057 +0000 UTC m=+428.587746826" watchObservedRunningTime="2026-09-09 23:55:16.381440346 +0000 UTC m=+428.588580114" Sep 09 23:55:16.383850 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.383833 2574 scope.go:117] "RemoveContainer" containerID="c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1" Sep 09 23:55:16.384128 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:16.384111 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1\": container with ID starting with c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1 not found: ID does not exist" containerID="c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1" Sep 09 23:55:16.384172 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.384137 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1"} err="failed to get container status \"c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1\": rpc error: code = NotFound desc = could not find container \"c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1\": container with ID starting with c68c7a2780691b0baf5854fab183894114406431e553077f241efcd64b3ac4c1 not found: ID does not exist" Sep 09 23:55:16.400071 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:16.400028 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" podStartSLOduration=1.400016431 podStartE2EDuration="1.400016431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:55:16.398894959 +0000 UTC m=+428.606034730" watchObservedRunningTime="2026-09-09 23:55:16.400016431 +0000 UTC m=+428.607156211" Sep 09 23:55:27.378265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:27.378236 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ncvpt" Sep 09 23:55:27.378736 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:27.378316 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:31.920574 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:31.920488 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn"] Sep 09 23:55:31.921014 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:31.920716 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" podUID="48aefcbb-ecd8-4408-95e4-dc018e1481eb" containerName="manager" containerID="cri-o://5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4" gracePeriod=10 Sep 09 23:55:32.154529 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.154497 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw"] Sep 09 23:55:32.168578 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.168554 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:55:32.170554 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.170532 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw"] Sep 09 23:55:32.181591 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.181571 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:32.278672 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.278642 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48aefcbb-ecd8-4408-95e4-dc018e1481eb-extensions-socket-volume\") pod \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\" (UID: \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\") " Sep 09 23:55:32.278835 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.278720 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78bzx\" (UniqueName: \"kubernetes.io/projected/48aefcbb-ecd8-4408-95e4-dc018e1481eb-kube-api-access-78bzx\") pod \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\" (UID: \"48aefcbb-ecd8-4408-95e4-dc018e1481eb\") " Sep 09 23:55:32.278927 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.278906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5g5w\" (UniqueName: \"kubernetes.io/projected/71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3-kube-api-access-x5g5w\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zd6fw\" (UID: \"71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:55:32.278999 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.278973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zd6fw\" (UID: \"71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:55:32.279046 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.278999 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48aefcbb-ecd8-4408-95e4-dc018e1481eb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "48aefcbb-ecd8-4408-95e4-dc018e1481eb" (UID: "48aefcbb-ecd8-4408-95e4-dc018e1481eb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:55:32.280857 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.280832 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48aefcbb-ecd8-4408-95e4-dc018e1481eb-kube-api-access-78bzx" (OuterVolumeSpecName: "kube-api-access-78bzx") pod "48aefcbb-ecd8-4408-95e4-dc018e1481eb" (UID: "48aefcbb-ecd8-4408-95e4-dc018e1481eb"). InnerVolumeSpecName "kube-api-access-78bzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:55:32.380318 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.380274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5g5w\" (UniqueName: \"kubernetes.io/projected/71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3-kube-api-access-x5g5w\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zd6fw\" (UID: \"71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:55:32.380445 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.380339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zd6fw\" (UID: \"71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:55:32.380445 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.380377 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78bzx\" (UniqueName: \"kubernetes.io/projected/48aefcbb-ecd8-4408-95e4-dc018e1481eb-kube-api-access-78bzx\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:55:32.380445 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.380387 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48aefcbb-ecd8-4408-95e4-dc018e1481eb-extensions-socket-volume\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:55:32.380666 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.380650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zd6fw\" (UID: \"71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:55:32.387990 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.387970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5g5w\" (UniqueName: \"kubernetes.io/projected/71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3-kube-api-access-x5g5w\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zd6fw\" (UID: \"71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:55:32.452720 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.452657 2574 generic.go:356] "Generic (PLEG): container finished" podID="48aefcbb-ecd8-4408-95e4-dc018e1481eb" containerID="5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4" exitCode=0 Sep 09 23:55:32.452720 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.452705 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" Sep 09 23:55:32.452720 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.452710 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" event={"ID":"48aefcbb-ecd8-4408-95e4-dc018e1481eb","Type":"ContainerDied","Data":"5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4"} Sep 09 23:55:32.452891 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.452761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn" event={"ID":"48aefcbb-ecd8-4408-95e4-dc018e1481eb","Type":"ContainerDied","Data":"2bd4b8a1dbdcc73c85f869c0a93e5ceaa13d564dc49ae0359626cef0ac00281a"} Sep 09 23:55:32.452891 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.452792 2574 scope.go:117] "RemoveContainer" containerID="5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4" Sep 09 23:55:32.461472 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.461455 2574 scope.go:117] "RemoveContainer" containerID="5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4" Sep 09 23:55:32.461779 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:55:32.461755 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4\": container with ID starting with 5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4 not found: ID does not exist" containerID="5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4" Sep 09 23:55:32.461840 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.461784 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4"} err="failed to get container status \"5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4\": rpc error: code = NotFound desc = could not find container \"5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4\": container with ID starting with 5659eec48644fbb5f2ab55e18e7d1dd9fa3d761ccde36c4de8bfe903f8393af4 not found: ID does not exist" Sep 09 23:55:32.474338 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.474313 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn"] Sep 09 23:55:32.480200 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.480180 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mhrbn"] Sep 09 23:55:32.480917 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.480900 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:55:32.775152 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:32.775122 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw"] Sep 09 23:55:32.777744 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:55:32.777720 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fac0f8_8e6e_42ce_8fd2_ac88f1c7bbc3.slice/crio-f230d9880b9584fa3be8f237c11a3c064574c7dc486ff1b7d414e8bd0352084c WatchSource:0}: Error finding container f230d9880b9584fa3be8f237c11a3c064574c7dc486ff1b7d414e8bd0352084c: Status 404 returned error can't find the container with id f230d9880b9584fa3be8f237c11a3c064574c7dc486ff1b7d414e8bd0352084c Sep 09 23:55:33.465812 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:33.465776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" event={"ID":"71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3","Type":"ContainerStarted","Data":"8c2ab8ea35cde58c00f5300caed6494591c62adbd62565751978e252858c05b0"} Sep 09 23:55:33.466208 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:33.465849 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" event={"ID":"71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3","Type":"ContainerStarted","Data":"f230d9880b9584fa3be8f237c11a3c064574c7dc486ff1b7d414e8bd0352084c"} Sep 09 23:55:33.466208 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:33.465926 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:55:33.489498 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:33.489452 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" podStartSLOduration=1.489439207 podStartE2EDuration="1.489439207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:55:33.486255383 +0000 UTC m=+445.693395153" watchObservedRunningTime="2026-09-09 23:55:33.489439207 +0000 UTC m=+445.696578992" Sep 09 23:55:34.239145 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:34.239111 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48aefcbb-ecd8-4408-95e4-dc018e1481eb" path="/var/lib/kubelet/pods/48aefcbb-ecd8-4408-95e4-dc018e1481eb/volumes" Sep 09 23:55:44.472858 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:55:44.472808 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zd6fw" Sep 09 23:56:03.175547 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.175515 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:56:03.175926 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.175842 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48aefcbb-ecd8-4408-95e4-dc018e1481eb" containerName="manager" Sep 09 23:56:03.175926 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.175852 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48aefcbb-ecd8-4408-95e4-dc018e1481eb" containerName="manager" Sep 09 23:56:03.175926 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.175919 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48aefcbb-ecd8-4408-95e4-dc018e1481eb" containerName="manager" Sep 09 23:56:03.178892 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.178866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:03.181314 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.181294 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 23:56:03.190431 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.190408 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:56:03.217131 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.217100 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:56:03.332915 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.332885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c452515c-63af-4481-8aa8-ef6207511964-config-file\") pod \"limitador-limitador-84994f7ffc-l8tqq\" (UID: \"c452515c-63af-4481-8aa8-ef6207511964\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:03.333366 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.333333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgmr\" (UniqueName: \"kubernetes.io/projected/c452515c-63af-4481-8aa8-ef6207511964-kube-api-access-7mgmr\") pod \"limitador-limitador-84994f7ffc-l8tqq\" (UID: \"c452515c-63af-4481-8aa8-ef6207511964\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:03.434232 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.434163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c452515c-63af-4481-8aa8-ef6207511964-config-file\") pod \"limitador-limitador-84994f7ffc-l8tqq\" (UID: \"c452515c-63af-4481-8aa8-ef6207511964\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:03.434232 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.434217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgmr\" (UniqueName: \"kubernetes.io/projected/c452515c-63af-4481-8aa8-ef6207511964-kube-api-access-7mgmr\") pod \"limitador-limitador-84994f7ffc-l8tqq\" (UID: \"c452515c-63af-4481-8aa8-ef6207511964\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:03.434775 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.434755 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c452515c-63af-4481-8aa8-ef6207511964-config-file\") pod \"limitador-limitador-84994f7ffc-l8tqq\" (UID: \"c452515c-63af-4481-8aa8-ef6207511964\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:03.441947 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.441919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgmr\" (UniqueName: \"kubernetes.io/projected/c452515c-63af-4481-8aa8-ef6207511964-kube-api-access-7mgmr\") pod \"limitador-limitador-84994f7ffc-l8tqq\" (UID: \"c452515c-63af-4481-8aa8-ef6207511964\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:03.489190 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.489164 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:03.589683 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.589654 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q6xhv"] Sep 09 23:56:03.593484 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.593463 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" Sep 09 23:56:03.595712 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.595694 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tj2zh\"" Sep 09 23:56:03.601314 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.601185 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q6xhv"] Sep 09 23:56:03.722537 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.722451 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-855kj"] Sep 09 23:56:03.725695 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.725672 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-855kj" Sep 09 23:56:03.732093 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.732071 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-855kj"] Sep 09 23:56:03.737096 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.737063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85p4\" (UniqueName: \"kubernetes.io/projected/52894f18-773b-4484-aeea-9a45341918a1-kube-api-access-s85p4\") pod \"authorino-7bcfd8f9cd-q6xhv\" (UID: \"52894f18-773b-4484-aeea-9a45341918a1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" Sep 09 23:56:03.759666 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.759639 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:56:03.761207 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:56:03.761174 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc452515c_63af_4481_8aa8_ef6207511964.slice/crio-74b145f4c2c38ca21ecc7eabff07607e044554317d35e4e52f49a56a9dba36c7 WatchSource:0}: Error finding container 74b145f4c2c38ca21ecc7eabff07607e044554317d35e4e52f49a56a9dba36c7: Status 404 returned error can't find the container with id 74b145f4c2c38ca21ecc7eabff07607e044554317d35e4e52f49a56a9dba36c7 Sep 09 23:56:03.837980 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.837957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85p4\" (UniqueName: \"kubernetes.io/projected/52894f18-773b-4484-aeea-9a45341918a1-kube-api-access-s85p4\") pod \"authorino-7bcfd8f9cd-q6xhv\" (UID: \"52894f18-773b-4484-aeea-9a45341918a1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" Sep 09 23:56:03.838082 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.838024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp62r\" (UniqueName: \"kubernetes.io/projected/5d62d782-503a-4823-a993-a98b09938c04-kube-api-access-rp62r\") pod \"authorino-59c974bd48-855kj\" (UID: \"5d62d782-503a-4823-a993-a98b09938c04\") " pod="kuadrant-system/authorino-59c974bd48-855kj" Sep 09 23:56:03.845606 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.845580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85p4\" (UniqueName: \"kubernetes.io/projected/52894f18-773b-4484-aeea-9a45341918a1-kube-api-access-s85p4\") pod \"authorino-7bcfd8f9cd-q6xhv\" (UID: \"52894f18-773b-4484-aeea-9a45341918a1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" Sep 09 23:56:03.903115 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.903091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" Sep 09 23:56:03.938707 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.938676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp62r\" (UniqueName: \"kubernetes.io/projected/5d62d782-503a-4823-a993-a98b09938c04-kube-api-access-rp62r\") pod \"authorino-59c974bd48-855kj\" (UID: \"5d62d782-503a-4823-a993-a98b09938c04\") " pod="kuadrant-system/authorino-59c974bd48-855kj" Sep 09 23:56:03.946543 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:03.946519 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp62r\" (UniqueName: \"kubernetes.io/projected/5d62d782-503a-4823-a993-a98b09938c04-kube-api-access-rp62r\") pod \"authorino-59c974bd48-855kj\" (UID: \"5d62d782-503a-4823-a993-a98b09938c04\") " pod="kuadrant-system/authorino-59c974bd48-855kj" Sep 09 23:56:04.035889 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:04.035815 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-855kj" Sep 09 23:56:04.169628 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:04.169601 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q6xhv"] Sep 09 23:56:04.170866 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:56:04.170843 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52894f18_773b_4484_aeea_9a45341918a1.slice/crio-ce5a8a90a5a09db86a317a094efeca944f32f5e88e2288b821c1df8bfadb03fe WatchSource:0}: Error finding container ce5a8a90a5a09db86a317a094efeca944f32f5e88e2288b821c1df8bfadb03fe: Status 404 returned error can't find the container with id ce5a8a90a5a09db86a317a094efeca944f32f5e88e2288b821c1df8bfadb03fe Sep 09 23:56:04.302174 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:04.302149 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-855kj"] Sep 09 23:56:04.303726 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:56:04.303698 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d62d782_503a_4823_a993_a98b09938c04.slice/crio-146df17308e493cf2bccd5093fb5c29f8f98523f341c8ce162307b93ef40cc78 WatchSource:0}: Error finding container 146df17308e493cf2bccd5093fb5c29f8f98523f341c8ce162307b93ef40cc78: Status 404 returned error can't find the container with id 146df17308e493cf2bccd5093fb5c29f8f98523f341c8ce162307b93ef40cc78 Sep 09 23:56:04.644335 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:04.644224 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" event={"ID":"c452515c-63af-4481-8aa8-ef6207511964","Type":"ContainerStarted","Data":"74b145f4c2c38ca21ecc7eabff07607e044554317d35e4e52f49a56a9dba36c7"} Sep 09 23:56:04.645722 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:04.645692 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-855kj" event={"ID":"5d62d782-503a-4823-a993-a98b09938c04","Type":"ContainerStarted","Data":"146df17308e493cf2bccd5093fb5c29f8f98523f341c8ce162307b93ef40cc78"} Sep 09 23:56:04.647130 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:04.647109 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" event={"ID":"52894f18-773b-4484-aeea-9a45341918a1","Type":"ContainerStarted","Data":"ce5a8a90a5a09db86a317a094efeca944f32f5e88e2288b821c1df8bfadb03fe"} Sep 09 23:56:05.656080 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:05.656043 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" event={"ID":"c452515c-63af-4481-8aa8-ef6207511964","Type":"ContainerStarted","Data":"2aed2f90732d2a86b65ae9cfdb0a5b7bdf57d74cd14ad3b039d1b51f793c4ef2"} Sep 09 23:56:05.656630 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:05.656209 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:05.673104 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:05.673050 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" podStartSLOduration=1.476834877 podStartE2EDuration="2.673035533s" totalImagesPullingTime="1.196200656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:56:05.671302648 +0000 UTC m=+477.878442418" watchObservedRunningTime="2026-09-09 23:56:05.673035533 +0000 UTC m=+477.880175305" Sep 09 23:56:06.663144 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:06.663104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-855kj" event={"ID":"5d62d782-503a-4823-a993-a98b09938c04","Type":"ContainerStarted","Data":"f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908"} Sep 09 23:56:06.664655 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:06.664632 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" event={"ID":"52894f18-773b-4484-aeea-9a45341918a1","Type":"ContainerStarted","Data":"2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0"} Sep 09 23:56:06.677995 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:06.677950 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-855kj" podStartSLOduration=1.9127915770000001 podStartE2EDuration="3.677937428s" totalImagesPullingTime="1.765145851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:56:06.677525935 +0000 UTC m=+478.884665704" watchObservedRunningTime="2026-09-09 23:56:06.677937428 +0000 UTC m=+478.885077196" Sep 09 23:56:06.689518 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:06.689453 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" podStartSLOduration=1.787045944 podStartE2EDuration="3.689436068s" totalImagesPullingTime="1.902390124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:56:06.688891806 +0000 UTC m=+478.896031575" watchObservedRunningTime="2026-09-09 23:56:06.689436068 +0000 UTC m=+478.896575840" Sep 09 23:56:06.714079 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:06.714046 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q6xhv"] Sep 09 23:56:08.675018 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:08.674979 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" podUID="52894f18-773b-4484-aeea-9a45341918a1" containerName="authorino" containerID="cri-o://2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0" gracePeriod=30 Sep 09 23:56:08.914419 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:08.914396 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" Sep 09 23:56:09.081052 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.081024 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s85p4\" (UniqueName: \"kubernetes.io/projected/52894f18-773b-4484-aeea-9a45341918a1-kube-api-access-s85p4\") pod \"52894f18-773b-4484-aeea-9a45341918a1\" (UID: \"52894f18-773b-4484-aeea-9a45341918a1\") " Sep 09 23:56:09.082943 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.082922 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52894f18-773b-4484-aeea-9a45341918a1-kube-api-access-s85p4" (OuterVolumeSpecName: "kube-api-access-s85p4") pod "52894f18-773b-4484-aeea-9a45341918a1" (UID: "52894f18-773b-4484-aeea-9a45341918a1"). InnerVolumeSpecName "kube-api-access-s85p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:56:09.182344 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.182315 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85p4\" (UniqueName: \"kubernetes.io/projected/52894f18-773b-4484-aeea-9a45341918a1-kube-api-access-s85p4\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:56:09.681973 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.681933 2574 generic.go:356] "Generic (PLEG): container finished" podID="52894f18-773b-4484-aeea-9a45341918a1" containerID="2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0" exitCode=0 Sep 09 23:56:09.681973 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.681963 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" Sep 09 23:56:09.682441 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.682012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" event={"ID":"52894f18-773b-4484-aeea-9a45341918a1","Type":"ContainerDied","Data":"2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0"} Sep 09 23:56:09.682441 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.682050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-q6xhv" event={"ID":"52894f18-773b-4484-aeea-9a45341918a1","Type":"ContainerDied","Data":"ce5a8a90a5a09db86a317a094efeca944f32f5e88e2288b821c1df8bfadb03fe"} Sep 09 23:56:09.682441 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.682068 2574 scope.go:117] "RemoveContainer" containerID="2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0" Sep 09 23:56:09.690839 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.690825 2574 scope.go:117] "RemoveContainer" containerID="2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0" Sep 09 23:56:09.691094 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:56:09.691074 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0\": container with ID starting with 2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0 not found: ID does not exist" containerID="2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0" Sep 09 23:56:09.691137 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.691100 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0"} err="failed to get container status \"2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0\": rpc error: code = NotFound desc = could not find container \"2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0\": container with ID starting with 2001ec175aa4922b93a087ab56a6807d1cbf1d6e490a277d9471c3e80f7d33c0 not found: ID does not exist" Sep 09 23:56:09.705458 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.705435 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q6xhv"] Sep 09 23:56:09.708875 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:09.708854 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q6xhv"] Sep 09 23:56:10.239466 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:10.239435 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52894f18-773b-4484-aeea-9a45341918a1" path="/var/lib/kubelet/pods/52894f18-773b-4484-aeea-9a45341918a1/volumes" Sep 09 23:56:16.665858 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:16.665830 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-l8tqq" Sep 09 23:56:33.720912 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.720879 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-4rn5h"] Sep 09 23:56:33.721416 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.721393 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52894f18-773b-4484-aeea-9a45341918a1" containerName="authorino" Sep 09 23:56:33.721493 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.721419 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="52894f18-773b-4484-aeea-9a45341918a1" containerName="authorino" Sep 09 23:56:33.721548 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.721536 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="52894f18-773b-4484-aeea-9a45341918a1" containerName="authorino" Sep 09 23:56:33.724240 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.724218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4rn5h" Sep 09 23:56:33.734845 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.734821 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-4rn5h"] Sep 09 23:56:33.763000 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.762977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4bz\" (UniqueName: \"kubernetes.io/projected/e75967b4-1361-475b-84dc-f1452b2c2f23-kube-api-access-rz4bz\") pod \"authorino-557f876f69-4rn5h\" (UID: \"e75967b4-1361-475b-84dc-f1452b2c2f23\") " pod="kuadrant-system/authorino-557f876f69-4rn5h" Sep 09 23:56:33.796055 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.796033 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4rn5h"] Sep 09 23:56:33.796255 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:56:33.796237 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rz4bz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-4rn5h" podUID="e75967b4-1361-475b-84dc-f1452b2c2f23" Sep 09 23:56:33.813488 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.813459 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4rn5h" Sep 09 23:56:33.819953 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.819928 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4rn5h" Sep 09 23:56:33.825787 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.825766 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d8t5c"] Sep 09 23:56:33.828816 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.828799 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:33.830988 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.830965 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 09 23:56:33.837444 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.837424 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d8t5c"] Sep 09 23:56:33.864015 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.863989 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrhf\" (UniqueName: \"kubernetes.io/projected/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-kube-api-access-2wrhf\") pod \"authorino-6d4dfc649f-d8t5c\" (UID: \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\") " pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:33.864127 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.864027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-tls-cert\") pod \"authorino-6d4dfc649f-d8t5c\" (UID: \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\") " pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:33.864127 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.864048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4bz\" (UniqueName: \"kubernetes.io/projected/e75967b4-1361-475b-84dc-f1452b2c2f23-kube-api-access-rz4bz\") pod \"authorino-557f876f69-4rn5h\" (UID: \"e75967b4-1361-475b-84dc-f1452b2c2f23\") " pod="kuadrant-system/authorino-557f876f69-4rn5h" Sep 09 23:56:33.872003 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.871980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4bz\" (UniqueName: \"kubernetes.io/projected/e75967b4-1361-475b-84dc-f1452b2c2f23-kube-api-access-rz4bz\") pod \"authorino-557f876f69-4rn5h\" (UID: \"e75967b4-1361-475b-84dc-f1452b2c2f23\") " pod="kuadrant-system/authorino-557f876f69-4rn5h" Sep 09 23:56:33.964502 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.964478 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz4bz\" (UniqueName: \"kubernetes.io/projected/e75967b4-1361-475b-84dc-f1452b2c2f23-kube-api-access-rz4bz\") pod \"e75967b4-1361-475b-84dc-f1452b2c2f23\" (UID: \"e75967b4-1361-475b-84dc-f1452b2c2f23\") " Sep 09 23:56:33.964629 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.964612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrhf\" (UniqueName: \"kubernetes.io/projected/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-kube-api-access-2wrhf\") pod \"authorino-6d4dfc649f-d8t5c\" (UID: \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\") " pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:33.964687 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.964648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-tls-cert\") pod \"authorino-6d4dfc649f-d8t5c\" (UID: \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\") " pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:33.966603 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.966580 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d8t5c"] Sep 09 23:56:33.966692 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.966594 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75967b4-1361-475b-84dc-f1452b2c2f23-kube-api-access-rz4bz" (OuterVolumeSpecName: "kube-api-access-rz4bz") pod "e75967b4-1361-475b-84dc-f1452b2c2f23" (UID: "e75967b4-1361-475b-84dc-f1452b2c2f23"). InnerVolumeSpecName "kube-api-access-rz4bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:56:33.966808 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:56:33.966790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2wrhf tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" podUID="bbf7f05c-00ac-494d-9a09-c7861ab87ba8" Sep 09 23:56:33.967069 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.967053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-tls-cert\") pod \"authorino-6d4dfc649f-d8t5c\" (UID: \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\") " pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:33.975425 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.975407 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrhf\" (UniqueName: \"kubernetes.io/projected/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-kube-api-access-2wrhf\") pod \"authorino-6d4dfc649f-d8t5c\" (UID: \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\") " pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:33.997497 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:33.997476 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c86fd5d56-p5bk7"] Sep 09 23:56:34.000726 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.000698 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" Sep 09 23:56:34.009226 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.009206 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c86fd5d56-p5bk7"] Sep 09 23:56:34.065138 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.065115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/68bac4bc-f14e-448b-982b-9ddf42359f61-tls-cert\") pod \"authorino-7c86fd5d56-p5bk7\" (UID: \"68bac4bc-f14e-448b-982b-9ddf42359f61\") " pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" Sep 09 23:56:34.065267 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.065143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5sdn\" (UniqueName: \"kubernetes.io/projected/68bac4bc-f14e-448b-982b-9ddf42359f61-kube-api-access-r5sdn\") pod \"authorino-7c86fd5d56-p5bk7\" (UID: \"68bac4bc-f14e-448b-982b-9ddf42359f61\") " pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" Sep 09 23:56:34.065340 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.065259 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4bz\" (UniqueName: \"kubernetes.io/projected/e75967b4-1361-475b-84dc-f1452b2c2f23-kube-api-access-rz4bz\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:56:34.166481 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.166454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/68bac4bc-f14e-448b-982b-9ddf42359f61-tls-cert\") pod \"authorino-7c86fd5d56-p5bk7\" (UID: \"68bac4bc-f14e-448b-982b-9ddf42359f61\") " pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" Sep 09 23:56:34.166621 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.166487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5sdn\" (UniqueName: \"kubernetes.io/projected/68bac4bc-f14e-448b-982b-9ddf42359f61-kube-api-access-r5sdn\") pod \"authorino-7c86fd5d56-p5bk7\" (UID: \"68bac4bc-f14e-448b-982b-9ddf42359f61\") " pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" Sep 09 23:56:34.168809 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.168781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/68bac4bc-f14e-448b-982b-9ddf42359f61-tls-cert\") pod \"authorino-7c86fd5d56-p5bk7\" (UID: \"68bac4bc-f14e-448b-982b-9ddf42359f61\") " pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" Sep 09 23:56:34.173985 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.173959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5sdn\" (UniqueName: \"kubernetes.io/projected/68bac4bc-f14e-448b-982b-9ddf42359f61-kube-api-access-r5sdn\") pod \"authorino-7c86fd5d56-p5bk7\" (UID: \"68bac4bc-f14e-448b-982b-9ddf42359f61\") " pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" Sep 09 23:56:34.311039 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.311017 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" Sep 09 23:56:34.576887 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.576858 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c86fd5d56-p5bk7"] Sep 09 23:56:34.578251 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:56:34.578226 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68bac4bc_f14e_448b_982b_9ddf42359f61.slice/crio-7de092eeb574bdaab59861ca698d2ceea5ce63f38d2debc2f62fad413400a215 WatchSource:0}: Error finding container 7de092eeb574bdaab59861ca698d2ceea5ce63f38d2debc2f62fad413400a215: Status 404 returned error can't find the container with id 7de092eeb574bdaab59861ca698d2ceea5ce63f38d2debc2f62fad413400a215 Sep 09 23:56:34.820318 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.820273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" event={"ID":"68bac4bc-f14e-448b-982b-9ddf42359f61","Type":"ContainerStarted","Data":"7de092eeb574bdaab59861ca698d2ceea5ce63f38d2debc2f62fad413400a215"} Sep 09 23:56:34.820318 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.820309 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:34.820318 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.820320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4rn5h" Sep 09 23:56:34.828762 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.828707 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:34.862917 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.862889 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4rn5h"] Sep 09 23:56:34.866775 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.866752 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-4rn5h"] Sep 09 23:56:34.872472 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.872453 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-tls-cert\") pod \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\" (UID: \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\") " Sep 09 23:56:34.872592 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.872498 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wrhf\" (UniqueName: \"kubernetes.io/projected/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-kube-api-access-2wrhf\") pod \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\" (UID: \"bbf7f05c-00ac-494d-9a09-c7861ab87ba8\") " Sep 09 23:56:34.874533 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.874505 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "bbf7f05c-00ac-494d-9a09-c7861ab87ba8" (UID: "bbf7f05c-00ac-494d-9a09-c7861ab87ba8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:56:34.874617 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.874532 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-kube-api-access-2wrhf" (OuterVolumeSpecName: "kube-api-access-2wrhf") pod "bbf7f05c-00ac-494d-9a09-c7861ab87ba8" (UID: "bbf7f05c-00ac-494d-9a09-c7861ab87ba8"). InnerVolumeSpecName "kube-api-access-2wrhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:56:34.973054 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.973029 2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-tls-cert\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:56:34.973054 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:34.973051 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrhf\" (UniqueName: \"kubernetes.io/projected/bbf7f05c-00ac-494d-9a09-c7861ab87ba8-kube-api-access-2wrhf\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:56:35.826675 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:35.826637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" event={"ID":"68bac4bc-f14e-448b-982b-9ddf42359f61","Type":"ContainerStarted","Data":"d17dfd6b14ef6a48757188b094cd4713b306fd90b0869f73316133adfad5bc90"} Sep 09 23:56:35.827115 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:35.826702 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d8t5c" Sep 09 23:56:35.843958 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:35.843911 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c86fd5d56-p5bk7" podStartSLOduration=2.246239007 podStartE2EDuration="2.843897558s" totalImagesPullingTime="597.658551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:56:35.841918881 +0000 UTC m=+508.049058653" watchObservedRunningTime="2026-09-09 23:56:35.843897558 +0000 UTC m=+508.051037327" Sep 09 23:56:35.864867 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:35.864842 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d8t5c"] Sep 09 23:56:35.868816 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:35.868791 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d8t5c"] Sep 09 23:56:35.872207 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:35.872185 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-855kj"] Sep 09 23:56:35.872378 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:35.872352 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-855kj" podUID="5d62d782-503a-4823-a993-a98b09938c04" containerName="authorino" containerID="cri-o://f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908" gracePeriod=30 Sep 09 23:56:36.120698 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.120677 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-855kj" Sep 09 23:56:36.183724 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.183699 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp62r\" (UniqueName: \"kubernetes.io/projected/5d62d782-503a-4823-a993-a98b09938c04-kube-api-access-rp62r\") pod \"5d62d782-503a-4823-a993-a98b09938c04\" (UID: \"5d62d782-503a-4823-a993-a98b09938c04\") " Sep 09 23:56:36.185751 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.185725 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d62d782-503a-4823-a993-a98b09938c04-kube-api-access-rp62r" (OuterVolumeSpecName: "kube-api-access-rp62r") pod "5d62d782-503a-4823-a993-a98b09938c04" (UID: "5d62d782-503a-4823-a993-a98b09938c04"). InnerVolumeSpecName "kube-api-access-rp62r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:56:36.240387 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.240362 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbf7f05c-00ac-494d-9a09-c7861ab87ba8" path="/var/lib/kubelet/pods/bbf7f05c-00ac-494d-9a09-c7861ab87ba8/volumes" Sep 09 23:56:36.240620 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.240608 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e75967b4-1361-475b-84dc-f1452b2c2f23" path="/var/lib/kubelet/pods/e75967b4-1361-475b-84dc-f1452b2c2f23/volumes" Sep 09 23:56:36.284386 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.284360 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp62r\" (UniqueName: \"kubernetes.io/projected/5d62d782-503a-4823-a993-a98b09938c04-kube-api-access-rp62r\") on node \"ip-10-0-129-102.ec2.internal\" DevicePath \"\"" Sep 09 23:56:36.833093 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.833054 2574 generic.go:356] "Generic (PLEG): container finished" podID="5d62d782-503a-4823-a993-a98b09938c04" containerID="f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908" exitCode=0 Sep 09 23:56:36.833598 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.833113 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-855kj" Sep 09 23:56:36.833598 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.833133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-855kj" event={"ID":"5d62d782-503a-4823-a993-a98b09938c04","Type":"ContainerDied","Data":"f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908"} Sep 09 23:56:36.833598 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.833178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-855kj" event={"ID":"5d62d782-503a-4823-a993-a98b09938c04","Type":"ContainerDied","Data":"146df17308e493cf2bccd5093fb5c29f8f98523f341c8ce162307b93ef40cc78"} Sep 09 23:56:36.833598 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.833193 2574 scope.go:117] "RemoveContainer" containerID="f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908" Sep 09 23:56:36.841582 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.841566 2574 scope.go:117] "RemoveContainer" containerID="f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908" Sep 09 23:56:36.841839 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:56:36.841821 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908\": container with ID starting with f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908 not found: ID does not exist" containerID="f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908" Sep 09 23:56:36.841913 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.841845 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908"} err="failed to get container status \"f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908\": rpc error: code = NotFound desc = could not find container \"f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908\": container with ID starting with f17bd2bb2911371f9b2224a88f91286bc567bbfb137be4c998fcd769dd838908 not found: ID does not exist" Sep 09 23:56:36.849068 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.849049 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-855kj"] Sep 09 23:56:36.854549 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:36.854528 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-855kj"] Sep 09 23:56:38.239847 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:56:38.239814 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d62d782-503a-4823-a993-a98b09938c04" path="/var/lib/kubelet/pods/5d62d782-503a-4823-a993-a98b09938c04/volumes" Sep 09 23:57:18.538250 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:18.538200 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:57:32.661642 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.661607 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4"] Sep 09 23:57:32.662113 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.662097 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d62d782-503a-4823-a993-a98b09938c04" containerName="authorino" Sep 09 23:57:32.662194 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.662116 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d62d782-503a-4823-a993-a98b09938c04" containerName="authorino" Sep 09 23:57:32.662256 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.662210 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d62d782-503a-4823-a993-a98b09938c04" containerName="authorino" Sep 09 23:57:32.665357 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.665333 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.667731 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.667699 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 23:57:32.668584 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.668569 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 23:57:32.668690 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.668604 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-shpnh\"" Sep 09 23:57:32.668778 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.668688 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 09 23:57:32.675778 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.675754 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4"] Sep 09 23:57:32.727170 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.727137 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.727170 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.727171 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.727360 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.727186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms8z5\" (UniqueName: \"kubernetes.io/projected/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-kube-api-access-ms8z5\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.727360 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.727207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.727360 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.727304 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.727458 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.727366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.827777 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.827749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.827911 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.827807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.827911 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.827837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.827911 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.827873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms8z5\" (UniqueName: \"kubernetes.io/projected/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-kube-api-access-ms8z5\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.827911 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.827904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.828121 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.827983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.828183 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.828136 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.828241 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.828203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.828359 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.828341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.830150 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.830127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.830367 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.830348 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.837620 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.837594 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms8z5\" (UniqueName: \"kubernetes.io/projected/bbe87c69-fda4-4a81-8a9b-e929cfd69be6-kube-api-access-ms8z5\") pod \"e2e-distinct-2-simulated-kserve-859856b874-4vgl4\" (UID: \"bbe87c69-fda4-4a81-8a9b-e929cfd69be6\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:32.971482 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.971395 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:57:32.978034 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:32.978008 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:33.255749 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:33.255718 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4"] Sep 09 23:57:33.257816 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:57:33.257781 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe87c69_fda4_4a81_8a9b_e929cfd69be6.slice/crio-daca923e224f0ee02a03a4d1466e720b15159849520509f5117c791ffaa649a3 WatchSource:0}: Error finding container daca923e224f0ee02a03a4d1466e720b15159849520509f5117c791ffaa649a3: Status 404 returned error can't find the container with id daca923e224f0ee02a03a4d1466e720b15159849520509f5117c791ffaa649a3 Sep 09 23:57:34.145636 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:34.145600 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" event={"ID":"bbe87c69-fda4-4a81-8a9b-e929cfd69be6","Type":"ContainerStarted","Data":"daca923e224f0ee02a03a4d1466e720b15159849520509f5117c791ffaa649a3"} Sep 09 23:57:35.152563 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:35.152529 2574 generic.go:356] "Generic (PLEG): container finished" podID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" containerID="3fcf64fa1cfbd87f291bdfd12bc68269bc0be96eec22b31b50a10b81c46c247d" exitCode=0 Sep 09 23:57:35.152956 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:35.152620 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" event={"ID":"bbe87c69-fda4-4a81-8a9b-e929cfd69be6","Type":"ContainerDied","Data":"3fcf64fa1cfbd87f291bdfd12bc68269bc0be96eec22b31b50a10b81c46c247d"} Sep 09 23:57:35.152956 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:35.152814 2574 scope.go:117] "RemoveContainer" containerID="3fcf64fa1cfbd87f291bdfd12bc68269bc0be96eec22b31b50a10b81c46c247d" Sep 09 23:57:36.159347 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:36.159312 2574 generic.go:356] "Generic (PLEG): container finished" podID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" containerID="590ac3f25edaba312a243b2ede80f6f74839d1294e75bcc98be716056cb8813a" exitCode=0 Sep 09 23:57:36.159745 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:36.159365 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" event={"ID":"bbe87c69-fda4-4a81-8a9b-e929cfd69be6","Type":"ContainerDied","Data":"590ac3f25edaba312a243b2ede80f6f74839d1294e75bcc98be716056cb8813a"} Sep 09 23:57:36.159745 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:36.159420 2574 scope.go:117] "RemoveContainer" containerID="3fcf64fa1cfbd87f291bdfd12bc68269bc0be96eec22b31b50a10b81c46c247d" Sep 09 23:57:36.159745 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:36.159641 2574 scope.go:117] "RemoveContainer" containerID="590ac3f25edaba312a243b2ede80f6f74839d1294e75bcc98be716056cb8813a" Sep 09 23:57:36.159880 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:57:36.159822 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:57:37.166802 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:37.166773 2574 scope.go:117] "RemoveContainer" containerID="590ac3f25edaba312a243b2ede80f6f74839d1294e75bcc98be716056cb8813a" Sep 09 23:57:37.167190 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:57:37.166940 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:57:41.564813 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:41.564771 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:57:42.978382 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:42.978335 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:42.978876 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:42.978414 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:42.978876 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:42.978434 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:42.978876 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:42.978456 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:57:42.979068 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:42.978948 2574 scope.go:117] "RemoveContainer" containerID="590ac3f25edaba312a243b2ede80f6f74839d1294e75bcc98be716056cb8813a" Sep 09 23:57:42.979181 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:57:42.979158 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:57:54.234233 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:54.234201 2574 scope.go:117] "RemoveContainer" containerID="590ac3f25edaba312a243b2ede80f6f74839d1294e75bcc98be716056cb8813a" Sep 09 23:57:55.263005 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.262825 2574 generic.go:356] "Generic (PLEG): container finished" podID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" containerID="ad88eb547f9824a28973b056efe65f27b6d91f8f9369bd4b557d5f4c53f5b4c7" exitCode=0 Sep 09 23:57:55.263005 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.262926 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" event={"ID":"bbe87c69-fda4-4a81-8a9b-e929cfd69be6","Type":"ContainerDied","Data":"ad88eb547f9824a28973b056efe65f27b6d91f8f9369bd4b557d5f4c53f5b4c7"} Sep 09 23:57:55.263005 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.262989 2574 scope.go:117] "RemoveContainer" containerID="590ac3f25edaba312a243b2ede80f6f74839d1294e75bcc98be716056cb8813a" Sep 09 23:57:55.263608 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.263403 2574 scope.go:117] "RemoveContainer" containerID="ad88eb547f9824a28973b056efe65f27b6d91f8f9369bd4b557d5f4c53f5b4c7" Sep 09 23:57:55.263670 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:57:55.263635 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:57:55.274053 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.274032 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk"] Sep 09 23:57:55.277487 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.277469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.279694 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.279676 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 09 23:57:55.288017 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.287993 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk"] Sep 09 23:57:55.414100 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.414062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgrnd\" (UniqueName: \"kubernetes.io/projected/cab297b3-99ac-412c-abb0-cc78b5cfcbca-kube-api-access-cgrnd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.414265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.414161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.414265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.414199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.414265 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.414228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.414395 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.414271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab297b3-99ac-412c-abb0-cc78b5cfcbca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.414395 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.414324 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.514834 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.514739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.514834 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.514777 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.514834 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.514799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.514834 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.514819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab297b3-99ac-412c-abb0-cc78b5cfcbca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.515186 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.514847 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.515186 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.514926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgrnd\" (UniqueName: \"kubernetes.io/projected/cab297b3-99ac-412c-abb0-cc78b5cfcbca-kube-api-access-cgrnd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.515328 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.515203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.515328 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.515310 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.515531 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.515493 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.517182 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.517158 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab297b3-99ac-412c-abb0-cc78b5cfcbca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.517345 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.517324 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab297b3-99ac-412c-abb0-cc78b5cfcbca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.521966 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.521942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgrnd\" (UniqueName: \"kubernetes.io/projected/cab297b3-99ac-412c-abb0-cc78b5cfcbca-kube-api-access-cgrnd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk\" (UID: \"cab297b3-99ac-412c-abb0-cc78b5cfcbca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.589057 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.589018 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:57:55.863054 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:55.863017 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk"] Sep 09 23:57:55.864621 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:57:55.864593 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab297b3_99ac_412c_abb0_cc78b5cfcbca.slice/crio-971e3d6f224d20906460e5232bd6453de0df3c6f904adfd07d88df00792b0592 WatchSource:0}: Error finding container 971e3d6f224d20906460e5232bd6453de0df3c6f904adfd07d88df00792b0592: Status 404 returned error can't find the container with id 971e3d6f224d20906460e5232bd6453de0df3c6f904adfd07d88df00792b0592 Sep 09 23:57:56.269116 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:56.269014 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" event={"ID":"cab297b3-99ac-412c-abb0-cc78b5cfcbca","Type":"ContainerStarted","Data":"5d43d34ac12fe076bdbb2489f6f10c811fbf6e2206e81f87bb4f6d256acec1c4"} Sep 09 23:57:56.269116 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:56.269070 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" event={"ID":"cab297b3-99ac-412c-abb0-cc78b5cfcbca","Type":"ContainerStarted","Data":"971e3d6f224d20906460e5232bd6453de0df3c6f904adfd07d88df00792b0592"} Sep 09 23:57:56.285982 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:56.285933 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" podStartSLOduration=1.117703214 podStartE2EDuration="1.285918097s" totalImagesPullingTime="168.214883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:57:56.283332931 +0000 UTC m=+588.490472699" watchObservedRunningTime="2026-09-09 23:57:56.285918097 +0000 UTC m=+588.493057925" Sep 09 23:57:56.464791 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:57:56.464753 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:58:02.978545 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:02.978511 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:58:02.978928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:02.978563 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:58:02.978928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:02.978579 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:58:02.978928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:02.978592 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:58:02.978928 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:02.978875 2574 scope.go:117] "RemoveContainer" containerID="ad88eb547f9824a28973b056efe65f27b6d91f8f9369bd4b557d5f4c53f5b4c7" Sep 09 23:58:02.979076 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:02.979030 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:58:05.589616 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:05.589586 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:58:05.590052 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:05.589678 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:58:05.602047 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:05.602027 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:58:06.333573 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:06.333548 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk" Sep 09 23:58:08.207650 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:08.207622 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 09 23:58:08.208159 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:08.208140 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 09 23:58:10.471196 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:10.471153 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:58:15.234028 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:15.233999 2574 scope.go:117] "RemoveContainer" containerID="ad88eb547f9824a28973b056efe65f27b6d91f8f9369bd4b557d5f4c53f5b4c7" Sep 09 23:58:15.236525 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:15.234250 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:58:16.377588 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:16.377557 2574 generic.go:356] "Generic (PLEG): container finished" podID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" containerID="d8e4efe75a00a077392c7dac60a510805d25eac4c85da60c0417e76dd9b4288c" exitCode=0 Sep 09 23:58:16.377980 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:16.377629 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" event={"ID":"bbe87c69-fda4-4a81-8a9b-e929cfd69be6","Type":"ContainerDied","Data":"d8e4efe75a00a077392c7dac60a510805d25eac4c85da60c0417e76dd9b4288c"} Sep 09 23:58:16.377980 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:16.377684 2574 scope.go:117] "RemoveContainer" containerID="ad88eb547f9824a28973b056efe65f27b6d91f8f9369bd4b557d5f4c53f5b4c7" Sep 09 23:58:16.378086 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:16.378068 2574 scope.go:117] "RemoveContainer" containerID="d8e4efe75a00a077392c7dac60a510805d25eac4c85da60c0417e76dd9b4288c" Sep 09 23:58:16.378250 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:16.378231 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:58:22.978529 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:22.978488 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:58:22.978947 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:22.978561 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:58:22.978947 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:22.978579 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:58:22.978947 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:22.978592 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:58:22.978947 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:22.978924 2574 scope.go:117] "RemoveContainer" containerID="d8e4efe75a00a077392c7dac60a510805d25eac4c85da60c0417e76dd9b4288c" Sep 09 23:58:22.979134 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:22.979113 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:58:25.478748 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:25.478711 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:58:28.555539 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.555506 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8"] Sep 09 23:58:28.560483 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.560459 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.562658 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.562633 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 09 23:58:28.569675 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.569646 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8"] Sep 09 23:58:28.687403 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.687375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.687555 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.687420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3214c6d5-8835-4e3f-b72b-3a37db599b7d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.687555 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.687442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89xhr\" (UniqueName: \"kubernetes.io/projected/3214c6d5-8835-4e3f-b72b-3a37db599b7d-kube-api-access-89xhr\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.687555 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.687514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.687555 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.687541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.687692 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.687564 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.788845 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.788812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.789008 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.788859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3214c6d5-8835-4e3f-b72b-3a37db599b7d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.789008 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.788880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89xhr\" (UniqueName: \"kubernetes.io/projected/3214c6d5-8835-4e3f-b72b-3a37db599b7d-kube-api-access-89xhr\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.789008 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.788918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.789008 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.788944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.789233 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.789087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.789233 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.789184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.789381 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.789249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.789381 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.789345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.791109 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.791091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3214c6d5-8835-4e3f-b72b-3a37db599b7d-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.791382 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.791364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3214c6d5-8835-4e3f-b72b-3a37db599b7d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.798518 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.798490 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89xhr\" (UniqueName: \"kubernetes.io/projected/3214c6d5-8835-4e3f-b72b-3a37db599b7d-kube-api-access-89xhr\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8\" (UID: \"3214c6d5-8835-4e3f-b72b-3a37db599b7d\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:28.871953 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:28.871892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:29.147020 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:29.146990 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8"] Sep 09 23:58:29.148536 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:58:29.148510 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3214c6d5_8835_4e3f_b72b_3a37db599b7d.slice/crio-8f51ecff0ea1ae9ab3d86d793b954af9d7af5a8b7326b94d81eb0720bb08e952 WatchSource:0}: Error finding container 8f51ecff0ea1ae9ab3d86d793b954af9d7af5a8b7326b94d81eb0720bb08e952: Status 404 returned error can't find the container with id 8f51ecff0ea1ae9ab3d86d793b954af9d7af5a8b7326b94d81eb0720bb08e952 Sep 09 23:58:29.448893 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:29.448866 2574 generic.go:356] "Generic (PLEG): container finished" podID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" containerID="7a546e81e5ffc5b3124bf7fa1eb71ad06442d5db7e43be4649f51b33dbf88ac5" exitCode=0 Sep 09 23:58:29.449001 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:29.448947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" event={"ID":"3214c6d5-8835-4e3f-b72b-3a37db599b7d","Type":"ContainerDied","Data":"7a546e81e5ffc5b3124bf7fa1eb71ad06442d5db7e43be4649f51b33dbf88ac5"} Sep 09 23:58:29.449044 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:29.449006 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" event={"ID":"3214c6d5-8835-4e3f-b72b-3a37db599b7d","Type":"ContainerStarted","Data":"8f51ecff0ea1ae9ab3d86d793b954af9d7af5a8b7326b94d81eb0720bb08e952"} Sep 09 23:58:29.449194 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:29.449181 2574 scope.go:117] "RemoveContainer" containerID="7a546e81e5ffc5b3124bf7fa1eb71ad06442d5db7e43be4649f51b33dbf88ac5" Sep 09 23:58:30.456426 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:30.456392 2574 generic.go:356] "Generic (PLEG): container finished" podID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" containerID="43c950de464d1e78c03190776bf30effe8fa3bac82c6ed51b4beeb593d36e592" exitCode=0 Sep 09 23:58:30.456945 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:30.456456 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" event={"ID":"3214c6d5-8835-4e3f-b72b-3a37db599b7d","Type":"ContainerDied","Data":"43c950de464d1e78c03190776bf30effe8fa3bac82c6ed51b4beeb593d36e592"} Sep 09 23:58:30.456945 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:30.456501 2574 scope.go:117] "RemoveContainer" containerID="7a546e81e5ffc5b3124bf7fa1eb71ad06442d5db7e43be4649f51b33dbf88ac5" Sep 09 23:58:30.456945 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:30.456795 2574 scope.go:117] "RemoveContainer" containerID="43c950de464d1e78c03190776bf30effe8fa3bac82c6ed51b4beeb593d36e592" Sep 09 23:58:30.457128 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:30.457011 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:58:30.571552 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:30.571515 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:58:31.463381 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:31.463354 2574 scope.go:117] "RemoveContainer" containerID="43c950de464d1e78c03190776bf30effe8fa3bac82c6ed51b4beeb593d36e592" Sep 09 23:58:31.463750 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:31.463527 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:58:35.233442 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:35.233410 2574 scope.go:117] "RemoveContainer" containerID="d8e4efe75a00a077392c7dac60a510805d25eac4c85da60c0417e76dd9b4288c" Sep 09 23:58:35.233807 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:35.233586 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:58:36.365396 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.365361 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw"] Sep 09 23:58:36.368952 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.368917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.371410 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.371381 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 09 23:58:36.379352 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.379328 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw"] Sep 09 23:58:36.454752 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.454724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.454752 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.454754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33277331-9456-4b26-ad88-aa235abb54d1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.454961 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.454788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-home\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.454961 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.454839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.454961 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.454853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.454961 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.454875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzmm\" (UniqueName: \"kubernetes.io/projected/33277331-9456-4b26-ad88-aa235abb54d1-kube-api-access-xtzmm\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.555362 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.555322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzmm\" (UniqueName: \"kubernetes.io/projected/33277331-9456-4b26-ad88-aa235abb54d1-kube-api-access-xtzmm\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.555546 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.555421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.555546 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.555447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33277331-9456-4b26-ad88-aa235abb54d1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.555546 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.555505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-home\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.555718 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.555568 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.555718 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.555591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.555946 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.555919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.556026 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.556004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-home\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.556108 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.556091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.558035 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.558004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33277331-9456-4b26-ad88-aa235abb54d1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.558138 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.558068 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33277331-9456-4b26-ad88-aa235abb54d1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.562599 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.562574 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzmm\" (UniqueName: \"kubernetes.io/projected/33277331-9456-4b26-ad88-aa235abb54d1-kube-api-access-xtzmm\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-b82mw\" (UID: \"33277331-9456-4b26-ad88-aa235abb54d1\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.678175 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.678106 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:36.950007 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:36.949981 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw"] Sep 09 23:58:36.951442 ip-10-0-129-102 kubenswrapper[2574]: W0909 23:58:36.951414 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33277331_9456_4b26_ad88_aa235abb54d1.slice/crio-91c8fa0d4e5b19a233a56bf657122f9d8ec9f809fbedc437179bc2e316724d12 WatchSource:0}: Error finding container 91c8fa0d4e5b19a233a56bf657122f9d8ec9f809fbedc437179bc2e316724d12: Status 404 returned error can't find the container with id 91c8fa0d4e5b19a233a56bf657122f9d8ec9f809fbedc437179bc2e316724d12 Sep 09 23:58:37.497461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:37.497365 2574 generic.go:356] "Generic (PLEG): container finished" podID="33277331-9456-4b26-ad88-aa235abb54d1" containerID="32b291f309aca8c1e2e01348c757e9f87b76bc566e607a5a0a2daed7ad027470" exitCode=0 Sep 09 23:58:37.497461 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:37.497450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" event={"ID":"33277331-9456-4b26-ad88-aa235abb54d1","Type":"ContainerDied","Data":"32b291f309aca8c1e2e01348c757e9f87b76bc566e607a5a0a2daed7ad027470"} Sep 09 23:58:37.497959 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:37.497488 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" event={"ID":"33277331-9456-4b26-ad88-aa235abb54d1","Type":"ContainerStarted","Data":"91c8fa0d4e5b19a233a56bf657122f9d8ec9f809fbedc437179bc2e316724d12"} Sep 09 23:58:37.497959 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:37.497732 2574 scope.go:117] "RemoveContainer" containerID="32b291f309aca8c1e2e01348c757e9f87b76bc566e607a5a0a2daed7ad027470" Sep 09 23:58:38.377811 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.377773 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-l8tqq"] Sep 09 23:58:38.505224 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.505194 2574 generic.go:356] "Generic (PLEG): container finished" podID="33277331-9456-4b26-ad88-aa235abb54d1" containerID="004378eabfa4038a99d3caedb3c36f0640785edc2b604aa5e806179b34df6eb5" exitCode=0 Sep 09 23:58:38.505626 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.505253 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" event={"ID":"33277331-9456-4b26-ad88-aa235abb54d1","Type":"ContainerDied","Data":"004378eabfa4038a99d3caedb3c36f0640785edc2b604aa5e806179b34df6eb5"} Sep 09 23:58:38.505626 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.505306 2574 scope.go:117] "RemoveContainer" containerID="32b291f309aca8c1e2e01348c757e9f87b76bc566e607a5a0a2daed7ad027470" Sep 09 23:58:38.505626 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.505518 2574 scope.go:117] "RemoveContainer" containerID="004378eabfa4038a99d3caedb3c36f0640785edc2b604aa5e806179b34df6eb5" Sep 09 23:58:38.505760 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:38.505700 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:58:38.872992 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.872955 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:38.873157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.873020 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:38.873157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.873035 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:38.873157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.873048 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:38.873442 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:38.873429 2574 scope.go:117] "RemoveContainer" containerID="43c950de464d1e78c03190776bf30effe8fa3bac82c6ed51b4beeb593d36e592" Sep 09 23:58:38.873612 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:38.873595 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:58:39.512247 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:39.512222 2574 scope.go:117] "RemoveContainer" containerID="004378eabfa4038a99d3caedb3c36f0640785edc2b604aa5e806179b34df6eb5" Sep 09 23:58:39.512663 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:39.512399 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:58:46.679093 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:46.679061 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:46.679093 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:46.679102 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:46.679620 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:46.679114 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:46.679620 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:46.679124 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:58:46.679620 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:46.679489 2574 scope.go:117] "RemoveContainer" containerID="004378eabfa4038a99d3caedb3c36f0640785edc2b604aa5e806179b34df6eb5" Sep 09 23:58:46.679716 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:46.679632 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:58:49.233559 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:49.233528 2574 scope.go:117] "RemoveContainer" containerID="d8e4efe75a00a077392c7dac60a510805d25eac4c85da60c0417e76dd9b4288c" Sep 09 23:58:49.233942 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:49.233716 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:58:50.233948 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:50.233916 2574 scope.go:117] "RemoveContainer" containerID="43c950de464d1e78c03190776bf30effe8fa3bac82c6ed51b4beeb593d36e592" Sep 09 23:58:50.577165 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:50.577140 2574 generic.go:356] "Generic (PLEG): container finished" podID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" containerID="c8a2abab0498edcda8f357590d268dfe673b159bdc30a823906450c52c7c377f" exitCode=0 Sep 09 23:58:50.577307 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:50.577215 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" event={"ID":"3214c6d5-8835-4e3f-b72b-3a37db599b7d","Type":"ContainerDied","Data":"c8a2abab0498edcda8f357590d268dfe673b159bdc30a823906450c52c7c377f"} Sep 09 23:58:50.577307 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:50.577269 2574 scope.go:117] "RemoveContainer" containerID="43c950de464d1e78c03190776bf30effe8fa3bac82c6ed51b4beeb593d36e592" Sep 09 23:58:50.577611 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:50.577587 2574 scope.go:117] "RemoveContainer" containerID="c8a2abab0498edcda8f357590d268dfe673b159bdc30a823906450c52c7c377f" Sep 09 23:58:50.577814 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:50.577786 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:58:58.238236 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.238209 2574 scope.go:117] "RemoveContainer" containerID="004378eabfa4038a99d3caedb3c36f0640785edc2b604aa5e806179b34df6eb5" Sep 09 23:58:58.623182 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.623147 2574 generic.go:356] "Generic (PLEG): container finished" podID="33277331-9456-4b26-ad88-aa235abb54d1" containerID="43bbe5b892f3aaca490cda27fc750f4b883ad2e5e0fd0d2b117c2a0e8d7c5eab" exitCode=0 Sep 09 23:58:58.623349 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.623224 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" event={"ID":"33277331-9456-4b26-ad88-aa235abb54d1","Type":"ContainerDied","Data":"43bbe5b892f3aaca490cda27fc750f4b883ad2e5e0fd0d2b117c2a0e8d7c5eab"} Sep 09 23:58:58.623349 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.623273 2574 scope.go:117] "RemoveContainer" containerID="004378eabfa4038a99d3caedb3c36f0640785edc2b604aa5e806179b34df6eb5" Sep 09 23:58:58.623614 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.623598 2574 scope.go:117] "RemoveContainer" containerID="43bbe5b892f3aaca490cda27fc750f4b883ad2e5e0fd0d2b117c2a0e8d7c5eab" Sep 09 23:58:58.623803 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:58.623784 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:58:58.872987 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.872950 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:58.873157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.873011 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:58.873157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.873027 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:58.873157 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.873040 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:58:58.873380 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:58:58.873365 2574 scope.go:117] "RemoveContainer" containerID="c8a2abab0498edcda8f357590d268dfe673b159bdc30a823906450c52c7c377f" Sep 09 23:58:58.873546 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:58:58.873529 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:59:03.234428 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:03.234391 2574 scope.go:117] "RemoveContainer" containerID="d8e4efe75a00a077392c7dac60a510805d25eac4c85da60c0417e76dd9b4288c" Sep 09 23:59:03.653087 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:03.653054 2574 generic.go:356] "Generic (PLEG): container finished" podID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" exitCode=0 Sep 09 23:59:03.653087 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:03.653090 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" event={"ID":"bbe87c69-fda4-4a81-8a9b-e929cfd69be6","Type":"ContainerDied","Data":"d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe"} Sep 09 23:59:03.653320 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:03.653138 2574 scope.go:117] "RemoveContainer" containerID="d8e4efe75a00a077392c7dac60a510805d25eac4c85da60c0417e76dd9b4288c" Sep 09 23:59:03.653527 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:03.653507 2574 scope.go:117] "RemoveContainer" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" Sep 09 23:59:03.653748 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:03.653729 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:59:06.678996 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:06.678967 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:59:06.679412 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:06.679009 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:59:06.679412 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:06.679020 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:59:06.679412 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:06.679030 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:59:06.679412 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:06.679392 2574 scope.go:117] "RemoveContainer" containerID="43bbe5b892f3aaca490cda27fc750f4b883ad2e5e0fd0d2b117c2a0e8d7c5eab" Sep 09 23:59:06.679573 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:06.679555 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:59:07.675118 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:07.675084 2574 scope.go:117] "RemoveContainer" containerID="43bbe5b892f3aaca490cda27fc750f4b883ad2e5e0fd0d2b117c2a0e8d7c5eab" Sep 09 23:59:07.675320 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:07.675248 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:59:12.233902 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.233868 2574 scope.go:117] "RemoveContainer" containerID="c8a2abab0498edcda8f357590d268dfe673b159bdc30a823906450c52c7c377f" Sep 09 23:59:12.702815 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.702783 2574 generic.go:356] "Generic (PLEG): container finished" podID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" containerID="375792149b16a1a3f9b6dd4cc4fa92aa677eceed8482a6a3a35b787ef12af367" exitCode=0 Sep 09 23:59:12.702983 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.702855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" event={"ID":"3214c6d5-8835-4e3f-b72b-3a37db599b7d","Type":"ContainerDied","Data":"375792149b16a1a3f9b6dd4cc4fa92aa677eceed8482a6a3a35b787ef12af367"} Sep 09 23:59:12.702983 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.702924 2574 scope.go:117] "RemoveContainer" containerID="c8a2abab0498edcda8f357590d268dfe673b159bdc30a823906450c52c7c377f" Sep 09 23:59:12.703269 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.703248 2574 scope.go:117] "RemoveContainer" containerID="375792149b16a1a3f9b6dd4cc4fa92aa677eceed8482a6a3a35b787ef12af367" Sep 09 23:59:12.703477 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:12.703454 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:59:12.978737 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.978700 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:59:12.978986 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.978771 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:59:12.978986 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.978791 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:59:12.978986 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.978810 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 09 23:59:12.979189 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:12.979170 2574 scope.go:117] "RemoveContainer" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" Sep 09 23:59:12.979390 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:12.979371 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:59:18.872229 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:18.872187 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:59:18.872799 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:18.872249 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:59:18.872799 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:18.872274 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:59:18.872799 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:18.872317 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 09 23:59:18.872799 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:18.872642 2574 scope.go:117] "RemoveContainer" containerID="375792149b16a1a3f9b6dd4cc4fa92aa677eceed8482a6a3a35b787ef12af367" Sep 09 23:59:18.872933 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:18.872830 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:59:19.233684 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:19.233608 2574 scope.go:117] "RemoveContainer" containerID="43bbe5b892f3aaca490cda27fc750f4b883ad2e5e0fd0d2b117c2a0e8d7c5eab" Sep 09 23:59:19.744084 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:19.744049 2574 generic.go:356] "Generic (PLEG): container finished" podID="33277331-9456-4b26-ad88-aa235abb54d1" containerID="87d9e3afd9caf65d5975d78bd286aa525da775a4923524ba324aece61ed26c1f" exitCode=0 Sep 09 23:59:19.744244 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:19.744113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" event={"ID":"33277331-9456-4b26-ad88-aa235abb54d1","Type":"ContainerDied","Data":"87d9e3afd9caf65d5975d78bd286aa525da775a4923524ba324aece61ed26c1f"} Sep 09 23:59:19.744244 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:19.744161 2574 scope.go:117] "RemoveContainer" containerID="43bbe5b892f3aaca490cda27fc750f4b883ad2e5e0fd0d2b117c2a0e8d7c5eab" Sep 09 23:59:19.744564 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:19.744544 2574 scope.go:117] "RemoveContainer" containerID="375792149b16a1a3f9b6dd4cc4fa92aa677eceed8482a6a3a35b787ef12af367" Sep 09 23:59:19.744702 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:19.744611 2574 scope.go:117] "RemoveContainer" containerID="87d9e3afd9caf65d5975d78bd286aa525da775a4923524ba324aece61ed26c1f" Sep 09 23:59:19.744794 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:19.744769 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:59:19.744871 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:19.744780 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:59:26.678587 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:26.678549 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:59:26.678587 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:26.678593 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:59:26.679112 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:26.678604 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:59:26.679112 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:26.678613 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 09 23:59:26.679112 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:26.679073 2574 scope.go:117] "RemoveContainer" containerID="87d9e3afd9caf65d5975d78bd286aa525da775a4923524ba324aece61ed26c1f" Sep 09 23:59:26.679277 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:26.679256 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:59:26.788387 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:26.788362 2574 scope.go:117] "RemoveContainer" containerID="87d9e3afd9caf65d5975d78bd286aa525da775a4923524ba324aece61ed26c1f" Sep 09 23:59:26.788541 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:26.788516 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:59:27.234537 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:27.234507 2574 scope.go:117] "RemoveContainer" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" Sep 09 23:59:27.234705 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:27.234664 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:59:32.233935 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:32.233904 2574 scope.go:117] "RemoveContainer" containerID="375792149b16a1a3f9b6dd4cc4fa92aa677eceed8482a6a3a35b787ef12af367" Sep 09 23:59:32.234331 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:32.234073 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:59:38.237208 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:38.237174 2574 scope.go:117] "RemoveContainer" containerID="87d9e3afd9caf65d5975d78bd286aa525da775a4923524ba324aece61ed26c1f" Sep 09 23:59:38.237606 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:38.237379 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:59:39.234404 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:39.234373 2574 scope.go:117] "RemoveContainer" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" Sep 09 23:59:39.234558 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:39.234537 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:59:45.234463 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:45.234427 2574 scope.go:117] "RemoveContainer" containerID="375792149b16a1a3f9b6dd4cc4fa92aa677eceed8482a6a3a35b787ef12af367" Sep 09 23:59:45.234846 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:45.234607 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 09 23:59:50.234196 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:50.234165 2574 scope.go:117] "RemoveContainer" containerID="87d9e3afd9caf65d5975d78bd286aa525da775a4923524ba324aece61ed26c1f" Sep 09 23:59:50.234697 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:50.234355 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 09 23:59:51.234067 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:51.234037 2574 scope.go:117] "RemoveContainer" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" Sep 09 23:59:51.234245 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:51.234226 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 09 23:59:58.236342 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:58.236316 2574 scope.go:117] "RemoveContainer" containerID="375792149b16a1a3f9b6dd4cc4fa92aa677eceed8482a6a3a35b787ef12af367" Sep 09 23:59:58.965995 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:58.965965 2574 generic.go:356] "Generic (PLEG): container finished" podID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" containerID="84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b" exitCode=0 Sep 09 23:59:58.966184 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:58.966044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" event={"ID":"3214c6d5-8835-4e3f-b72b-3a37db599b7d","Type":"ContainerDied","Data":"84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b"} Sep 09 23:59:58.966184 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:58.966120 2574 scope.go:117] "RemoveContainer" containerID="375792149b16a1a3f9b6dd4cc4fa92aa677eceed8482a6a3a35b787ef12af367" Sep 09 23:59:58.966505 ip-10-0-129-102 kubenswrapper[2574]: I0909 23:59:58.966490 2574 scope.go:117] "RemoveContainer" containerID="84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b" Sep 09 23:59:58.966694 ip-10-0-129-102 kubenswrapper[2574]: E0909 23:59:58.966676 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:00:04.234065 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:04.233988 2574 scope.go:117] "RemoveContainer" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" Sep 10 00:00:04.234447 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:04.234163 2574 scope.go:117] "RemoveContainer" containerID="87d9e3afd9caf65d5975d78bd286aa525da775a4923524ba324aece61ed26c1f" Sep 10 00:00:04.234447 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:04.234207 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:00:07.013588 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:07.013558 2574 generic.go:356] "Generic (PLEG): container finished" podID="33277331-9456-4b26-ad88-aa235abb54d1" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" exitCode=0 Sep 10 00:00:07.013957 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:07.013630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" event={"ID":"33277331-9456-4b26-ad88-aa235abb54d1","Type":"ContainerDied","Data":"bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d"} Sep 10 00:00:07.013957 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:07.013680 2574 scope.go:117] "RemoveContainer" containerID="87d9e3afd9caf65d5975d78bd286aa525da775a4923524ba324aece61ed26c1f" Sep 10 00:00:07.014047 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:07.014029 2574 scope.go:117] "RemoveContainer" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" Sep 10 00:00:07.014214 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:07.014193 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:00:08.872262 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:08.872212 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 10 00:00:08.872677 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:08.872310 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 10 00:00:08.872677 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:08.872328 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 10 00:00:08.872677 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:08.872340 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 10 00:00:08.872677 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:08.872667 2574 scope.go:117] "RemoveContainer" containerID="84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b" Sep 10 00:00:08.872844 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:08.872826 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:00:16.679059 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:16.679023 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 10 00:00:16.679059 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:16.679069 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 10 00:00:16.679502 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:16.679082 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 10 00:00:16.679502 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:16.679095 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 10 00:00:16.679502 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:16.679468 2574 scope.go:117] "RemoveContainer" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" Sep 10 00:00:16.679644 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:16.679625 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:00:18.241056 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:18.241022 2574 scope.go:117] "RemoveContainer" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" Sep 10 00:00:18.241867 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:18.241831 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:00:24.234196 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:24.234161 2574 scope.go:117] "RemoveContainer" containerID="84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b" Sep 10 00:00:24.234701 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:24.234367 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:00:29.233751 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:29.233720 2574 scope.go:117] "RemoveContainer" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" Sep 10 00:00:29.234131 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:29.233875 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:00:33.233439 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:33.233404 2574 scope.go:117] "RemoveContainer" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" Sep 10 00:00:34.162980 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:34.162947 2574 generic.go:356] "Generic (PLEG): container finished" podID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" exitCode=0 Sep 10 00:00:34.163187 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:34.163020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" event={"ID":"bbe87c69-fda4-4a81-8a9b-e929cfd69be6","Type":"ContainerDied","Data":"20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614"} Sep 10 00:00:34.163187 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:34.163080 2574 scope.go:117] "RemoveContainer" containerID="d0271580708d13ba29b94da86c087dc501622bc683b430c6bf0844ef77cb55fe" Sep 10 00:00:34.163580 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:34.163561 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:00:34.163849 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:34.163827 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:00:38.237144 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:38.237115 2574 scope.go:117] "RemoveContainer" containerID="84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b" Sep 10 00:00:38.237545 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:38.237330 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:00:42.233699 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:42.233669 2574 scope.go:117] "RemoveContainer" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" Sep 10 00:00:42.234116 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:42.233832 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:00:42.978139 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:42.978108 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 10 00:00:42.978326 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:42.978158 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 10 00:00:42.978326 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:42.978173 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 10 00:00:42.978326 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:42.978187 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" Sep 10 00:00:42.978516 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:42.978503 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:00:42.978682 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:42.978663 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:00:52.234112 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:52.234082 2574 scope.go:117] "RemoveContainer" containerID="84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b" Sep 10 00:00:52.234510 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:52.234255 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:00:56.234211 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:56.234181 2574 scope.go:117] "RemoveContainer" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" Sep 10 00:00:56.234715 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:56.234365 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:00:58.236770 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:00:58.236744 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:00:58.237170 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:00:58.236914 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:01:07.233758 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:07.233721 2574 scope.go:117] "RemoveContainer" containerID="84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b" Sep 10 00:01:07.234143 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:07.233907 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:01:09.233484 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:09.233444 2574 scope.go:117] "RemoveContainer" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" Sep 10 00:01:09.233922 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:09.233624 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:01:09.233922 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:09.233635 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:01:09.233922 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:09.233790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:01:20.234073 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:20.234040 2574 scope.go:117] "RemoveContainer" containerID="84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b" Sep 10 00:01:21.233820 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:21.233785 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:01:21.233999 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:21.233965 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:01:21.430139 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:21.430103 2574 generic.go:356] "Generic (PLEG): container finished" podID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" exitCode=0 Sep 10 00:01:21.430559 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:21.430173 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" event={"ID":"3214c6d5-8835-4e3f-b72b-3a37db599b7d","Type":"ContainerDied","Data":"2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221"} Sep 10 00:01:21.430559 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:21.430238 2574 scope.go:117] "RemoveContainer" containerID="84dad7b7620812cff841a5656c4ae1f9658f35b1690df7e6d48aed55cfadeb4b" Sep 10 00:01:21.430691 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:21.430606 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:01:21.430820 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:21.430799 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:01:24.233627 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:24.233591 2574 scope.go:117] "RemoveContainer" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" Sep 10 00:01:24.234011 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:24.233744 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:01:28.872326 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:28.872293 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 10 00:01:28.872794 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:28.872353 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 10 00:01:28.872794 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:28.872372 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 10 00:01:28.872794 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:28.872386 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" Sep 10 00:01:28.872794 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:28.872725 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:01:28.872944 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:28.872923 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:01:33.233845 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:33.233763 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:01:33.234213 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:33.233935 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:01:35.233487 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:35.233458 2574 scope.go:117] "RemoveContainer" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" Sep 10 00:01:35.517521 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:35.517488 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" event={"ID":"33277331-9456-4b26-ad88-aa235abb54d1","Type":"ContainerStarted","Data":"eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c"} Sep 10 00:01:35.535075 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:35.535030 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podStartSLOduration=176.473314111 podStartE2EDuration="2m59.535014852s" totalImagesPullingTime="3.061700741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:58:36 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:01:35.532527914 +0000 UTC m=+807.739667685" watchObservedRunningTime="2026-09-10 00:01:35.535014852 +0000 UTC m=+807.742154674" Sep 10 00:01:36.525434 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:36.525399 2574 generic.go:356] "Generic (PLEG): container finished" podID="33277331-9456-4b26-ad88-aa235abb54d1" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" exitCode=0 Sep 10 00:01:36.525909 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:36.525469 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" event={"ID":"33277331-9456-4b26-ad88-aa235abb54d1","Type":"ContainerDied","Data":"eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c"} Sep 10 00:01:36.525909 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:36.525519 2574 scope.go:117] "RemoveContainer" containerID="bf1f01206814a103eb4c7e0ae57a63cf266abd139d378c53fe1f9bbbbbc83d0d" Sep 10 00:01:36.525909 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:36.525835 2574 scope.go:117] "RemoveContainer" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" Sep 10 00:01:36.526085 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:36.526030 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:01:36.678978 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:36.678946 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 10 00:01:36.679153 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:36.678996 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 10 00:01:36.679153 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:36.679010 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 10 00:01:36.679153 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:36.679023 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" Sep 10 00:01:37.532604 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:37.532571 2574 scope.go:117] "RemoveContainer" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" Sep 10 00:01:37.533058 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:37.532770 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:01:42.234180 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:42.234147 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:01:42.234601 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:42.234346 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:01:44.233870 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:44.233840 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:01:44.234241 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:44.234014 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:01:51.234155 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:51.234119 2574 scope.go:117] "RemoveContainer" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" Sep 10 00:01:51.234568 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:51.234318 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:01:53.233782 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:53.233749 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:01:53.234167 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:53.233937 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:01:58.238412 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:01:58.238380 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:01:58.238863 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:01:58.238668 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:02:04.234183 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:04.234146 2574 scope.go:117] "RemoveContainer" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" Sep 10 00:02:04.234801 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:04.234403 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:02:06.234078 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:06.234046 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:02:06.234481 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:06.234264 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:02:13.234256 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:13.234223 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:02:13.234669 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:13.234442 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:02:16.234550 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:16.234521 2574 scope.go:117] "RemoveContainer" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" Sep 10 00:02:16.234926 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:16.234647 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:02:20.234035 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:20.234006 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:02:20.234514 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:20.234225 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:02:25.233614 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:25.233584 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:02:25.234014 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:25.233754 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:02:29.233780 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:29.233743 2574 scope.go:117] "RemoveContainer" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" Sep 10 00:02:29.234243 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:29.233953 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:02:31.349132 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:31.349101 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_3214c6d5-8835-4e3f-b72b-3a37db599b7d/main/5.log" Sep 10 00:02:31.461515 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:31.461493 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_3214c6d5-8835-4e3f-b72b-3a37db599b7d/main/5.log" Sep 10 00:02:32.602771 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:32.602740 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c86fd5d56-p5bk7_68bac4bc-f14e-448b-982b-9ddf42359f61/authorino/0.log" Sep 10 00:02:33.233751 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:33.233721 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:02:33.233941 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:33.233921 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:02:37.773349 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:37.773324 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pkslh_25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4/manager/0.log" Sep 10 00:02:37.889981 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:37.889952 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jm546_fce0520e-d5e7-4e3f-9532-a1765adffbad/manager/0.log" Sep 10 00:02:38.371562 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:38.371535 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7547cdb9fd-hdptp_2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545/copy-manifests/0.log" Sep 10 00:02:38.383130 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:38.383091 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7547cdb9fd-hdptp_2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545/manager/0.log" Sep 10 00:02:40.019222 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:40.019193 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c86fd5d56-p5bk7_68bac4bc-f14e-448b-982b-9ddf42359f61/authorino/0.log" Sep 10 00:02:40.133268 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:40.133239 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-97plq_54cbf93f-8a00-4ebd-9e63-61215a8ddfde/manager/0.log" Sep 10 00:02:40.233620 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:40.233591 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:02:40.233786 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:40.233748 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:02:40.246603 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:40.246581 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-w58fq_72e5a4e2-a0d3-4521-a090-d28f7d649dc4/manager/0.log" Sep 10 00:02:40.360976 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:40.360954 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-mc97c_e75f2bc6-cc62-4eed-86ee-4baad44a3a8f/kuadrant-console-plugin/0.log" Sep 10 00:02:40.480955 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:40.480932 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-zd6fw_71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3/manager/0.log" Sep 10 00:02:40.596630 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:40.596606 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-l8tqq_c452515c-63af-4481-8aa8-ef6207511964/limitador/0.log" Sep 10 00:02:40.712841 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:40.712781 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-ncvpt_b763f4e5-4255-4227-b319-858a759cfc4d/manager/0.log" Sep 10 00:02:41.416645 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:41.416620 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-947759788-x4q2t_e78d58b9-4f18-4f8a-bf6e-7c86427779f2/kube-auth-proxy/0.log" Sep 10 00:02:42.237162 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:42.237135 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-859856b874-4vgl4_bbe87c69-fda4-4a81-8a9b-e929cfd69be6/main/5.log" Sep 10 00:02:42.464763 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:42.464735 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-rbttk_cab297b3-99ac-412c-abb0-cc78b5cfcbca/main/0.log" Sep 10 00:02:42.814958 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:42.814928 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_3214c6d5-8835-4e3f-b72b-3a37db599b7d/main/5.log" Sep 10 00:02:42.926925 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:42.926889 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_33277331-9456-4b26-ad88-aa235abb54d1/main/5.log" Sep 10 00:02:44.234457 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:44.234431 2574 scope.go:117] "RemoveContainer" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" Sep 10 00:02:44.234926 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:44.234627 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:02:48.236424 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.236398 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:02:48.236855 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:48.236562 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:02:48.743147 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.743109 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6zkw7/must-gather-zksvs"] Sep 10 00:02:48.746984 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.746962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zkw7/must-gather-zksvs" Sep 10 00:02:48.749253 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.749233 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6zkw7\"/\"default-dockercfg-rgfzf\"" Sep 10 00:02:48.749551 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.749532 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6zkw7\"/\"openshift-service-ca.crt\"" Sep 10 00:02:48.750480 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.750457 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6zkw7\"/\"kube-root-ca.crt\"" Sep 10 00:02:48.757347 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.757323 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zkw7/must-gather-zksvs"] Sep 10 00:02:48.829862 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.829826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc820d15-1923-483c-9db6-3920a3c057b1-must-gather-output\") pod \"must-gather-zksvs\" (UID: \"dc820d15-1923-483c-9db6-3920a3c057b1\") " pod="openshift-must-gather-6zkw7/must-gather-zksvs" Sep 10 00:02:48.830011 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.829889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2bc\" (UniqueName: \"kubernetes.io/projected/dc820d15-1923-483c-9db6-3920a3c057b1-kube-api-access-4r2bc\") pod \"must-gather-zksvs\" (UID: \"dc820d15-1923-483c-9db6-3920a3c057b1\") " pod="openshift-must-gather-6zkw7/must-gather-zksvs" Sep 10 00:02:48.930695 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.930665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2bc\" (UniqueName: \"kubernetes.io/projected/dc820d15-1923-483c-9db6-3920a3c057b1-kube-api-access-4r2bc\") pod \"must-gather-zksvs\" (UID: \"dc820d15-1923-483c-9db6-3920a3c057b1\") " pod="openshift-must-gather-6zkw7/must-gather-zksvs" Sep 10 00:02:48.930837 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.930739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc820d15-1923-483c-9db6-3920a3c057b1-must-gather-output\") pod \"must-gather-zksvs\" (UID: \"dc820d15-1923-483c-9db6-3920a3c057b1\") " pod="openshift-must-gather-6zkw7/must-gather-zksvs" Sep 10 00:02:48.931044 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.931028 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc820d15-1923-483c-9db6-3920a3c057b1-must-gather-output\") pod \"must-gather-zksvs\" (UID: \"dc820d15-1923-483c-9db6-3920a3c057b1\") " pod="openshift-must-gather-6zkw7/must-gather-zksvs" Sep 10 00:02:48.939127 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:48.939108 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2bc\" (UniqueName: \"kubernetes.io/projected/dc820d15-1923-483c-9db6-3920a3c057b1-kube-api-access-4r2bc\") pod \"must-gather-zksvs\" (UID: \"dc820d15-1923-483c-9db6-3920a3c057b1\") " pod="openshift-must-gather-6zkw7/must-gather-zksvs" Sep 10 00:02:49.055983 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:49.055962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zkw7/must-gather-zksvs" Sep 10 00:02:49.326133 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:49.326094 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zkw7/must-gather-zksvs"] Sep 10 00:02:49.326761 ip-10-0-129-102 kubenswrapper[2574]: W0910 00:02:49.326734 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc820d15_1923_483c_9db6_3920a3c057b1.slice/crio-2368b053fc213373eef960717b96525d57e90c7d14204b583111a9a57a1e9a0e WatchSource:0}: Error finding container 2368b053fc213373eef960717b96525d57e90c7d14204b583111a9a57a1e9a0e: Status 404 returned error can't find the container with id 2368b053fc213373eef960717b96525d57e90c7d14204b583111a9a57a1e9a0e Sep 10 00:02:49.931627 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:49.931596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zkw7/must-gather-zksvs" event={"ID":"dc820d15-1923-483c-9db6-3920a3c057b1","Type":"ContainerStarted","Data":"2368b053fc213373eef960717b96525d57e90c7d14204b583111a9a57a1e9a0e"} Sep 10 00:02:50.943564 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:50.943527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zkw7/must-gather-zksvs" event={"ID":"dc820d15-1923-483c-9db6-3920a3c057b1","Type":"ContainerStarted","Data":"e5c9c5e735e18e0d8052d4fcfa85698aab2152f5050794b69537ba4fc4580888"} Sep 10 00:02:50.944104 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:50.944077 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zkw7/must-gather-zksvs" event={"ID":"dc820d15-1923-483c-9db6-3920a3c057b1","Type":"ContainerStarted","Data":"d3376c2b86fe94cb3953b0b56d413d56f86d72df493be6ed8c9c56f6db978744"} Sep 10 00:02:50.958905 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:50.958845 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6zkw7/must-gather-zksvs" podStartSLOduration=2.081130443 podStartE2EDuration="2.958826193s" totalImagesPullingTime="877.69575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:02:50.95761879 +0000 UTC m=+883.164758560" watchObservedRunningTime="2026-09-10 00:02:50.958826193 +0000 UTC m=+883.165965965" Sep 10 00:02:51.721575 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:51.721543 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bjngb_55083d6e-c367-40eb-96ad-b70390fc1039/global-pull-secret-syncer/0.log" Sep 10 00:02:51.853104 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:51.853074 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tfn7k_9847e1ea-9099-4ae4-a465-50c6a5d86acb/konnectivity-agent/0.log" Sep 10 00:02:51.873039 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:51.872996 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-102.ec2.internal_ef2e7d00f6cdedb8b1c2902e991acc2b/haproxy/0.log" Sep 10 00:02:54.246002 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:54.245537 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:02:54.246002 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:54.245786 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:02:56.243876 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:56.243844 2574 scope.go:117] "RemoveContainer" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" Sep 10 00:02:56.244476 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:02:56.244066 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:02:56.507701 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:56.507625 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c86fd5d56-p5bk7_68bac4bc-f14e-448b-982b-9ddf42359f61/authorino/0.log" Sep 10 00:02:56.547503 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:56.547459 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-97plq_54cbf93f-8a00-4ebd-9e63-61215a8ddfde/manager/0.log" Sep 10 00:02:56.574796 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:56.574748 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-w58fq_72e5a4e2-a0d3-4521-a090-d28f7d649dc4/manager/0.log" Sep 10 00:02:56.598769 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:56.598731 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-mc97c_e75f2bc6-cc62-4eed-86ee-4baad44a3a8f/kuadrant-console-plugin/0.log" Sep 10 00:02:56.633762 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:56.633730 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-zd6fw_71fac0f8-8e6e-42ce-8fd2-ac88f1c7bbc3/manager/0.log" Sep 10 00:02:56.658581 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:56.658556 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-l8tqq_c452515c-63af-4481-8aa8-ef6207511964/limitador/0.log" Sep 10 00:02:56.694033 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:56.693998 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-ncvpt_b763f4e5-4255-4227-b319-858a759cfc4d/manager/0.log" Sep 10 00:02:58.474345 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:58.474311 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-587c665f98-fhqhx_a9a255bc-472a-46da-9df5-2a0277aa6369/metrics-server/0.log" Sep 10 00:02:58.510392 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:58.510365 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-kxbpp_4fb6029b-3a6f-42ed-9b2e-c23e118e5ca6/monitoring-plugin/0.log" Sep 10 00:02:58.542946 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:58.542909 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f6tjk_c77e3ae3-38bf-4e62-9a5c-c690eef779d7/node-exporter/0.log" Sep 10 00:02:58.563258 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:58.563230 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f6tjk_c77e3ae3-38bf-4e62-9a5c-c690eef779d7/kube-rbac-proxy/0.log" Sep 10 00:02:58.585856 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:58.585829 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f6tjk_c77e3ae3-38bf-4e62-9a5c-c690eef779d7/init-textfile/0.log" Sep 10 00:02:58.769779 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:58.768042 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gc7z6_7061de57-0809-471e-9535-548e8218cce4/kube-rbac-proxy-main/0.log" Sep 10 00:02:58.792307 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:58.791109 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gc7z6_7061de57-0809-471e-9535-548e8218cce4/kube-rbac-proxy-self/0.log" Sep 10 00:02:58.827500 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:58.827470 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gc7z6_7061de57-0809-471e-9535-548e8218cce4/openshift-state-metrics/0.log" Sep 10 00:02:59.096093 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:59.096065 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b5787994-8f97s_4305d5fc-da8f-4379-805c-b56c2fb77df1/telemeter-client/0.log" Sep 10 00:02:59.122087 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:59.122055 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b5787994-8f97s_4305d5fc-da8f-4379-805c-b56c2fb77df1/reload/0.log" Sep 10 00:02:59.145002 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:02:59.144976 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b5787994-8f97s_4305d5fc-da8f-4379-805c-b56c2fb77df1/kube-rbac-proxy/0.log" Sep 10 00:03:01.629058 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:01.628985 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6dc545d574-2nlvs_5866a017-4f1f-4e48-afb8-7475194eb0fc/console/0.log" Sep 10 00:03:01.656536 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:01.656493 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-jfvwq_454960f3-186d-4d90-8801-ff88241088b9/download-server/0.log" Sep 10 00:03:02.317334 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.317274 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg"] Sep 10 00:03:02.323957 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.323930 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.339055 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.338933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg"] Sep 10 00:03:02.390827 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.390794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-sys\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.390827 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.390835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-lib-modules\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.391089 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.390859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-podres\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.391089 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.390936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-proc\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.391089 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.391009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5s4p\" (UniqueName: \"kubernetes.io/projected/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-kube-api-access-g5s4p\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.492664 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.492617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-sys\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.492856 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.492676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-lib-modules\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.492856 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.492702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-podres\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.492856 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.492729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-proc\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.492856 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.492757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5s4p\" (UniqueName: \"kubernetes.io/projected/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-kube-api-access-g5s4p\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.493212 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.493171 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-lib-modules\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.493212 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.493196 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-proc\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.493366 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.493245 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-sys\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.493522 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.493489 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-podres\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.500517 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.500492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5s4p\" (UniqueName: \"kubernetes.io/projected/5f1da6c5-5a69-4862-a8e5-4c91d591c5cd-kube-api-access-g5s4p\") pod \"perf-node-gather-daemonset-q4kpg\" (UID: \"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd\") " pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.664956 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.664869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:02.973379 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:02.973352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg"] Sep 10 00:03:02.976247 ip-10-0-129-102 kubenswrapper[2574]: W0910 00:03:02.976218 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5f1da6c5_5a69_4862_a8e5_4c91d591c5cd.slice/crio-022ddafd9b014b1d5fe5c543fc9e2fd709c00dd083d67ca057ded7a55a870997 WatchSource:0}: Error finding container 022ddafd9b014b1d5fe5c543fc9e2fd709c00dd083d67ca057ded7a55a870997: Status 404 returned error can't find the container with id 022ddafd9b014b1d5fe5c543fc9e2fd709c00dd083d67ca057ded7a55a870997 Sep 10 00:03:03.046315 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:03.046268 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" event={"ID":"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd","Type":"ContainerStarted","Data":"022ddafd9b014b1d5fe5c543fc9e2fd709c00dd083d67ca057ded7a55a870997"} Sep 10 00:03:03.233833 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:03.233753 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:03:03.234018 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:03:03.233989 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:03:03.408575 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:03.408536 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4952r_3e147845-f357-4d4e-a183-c8a22727e954/dns/0.log" Sep 10 00:03:03.429263 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:03.429214 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4952r_3e147845-f357-4d4e-a183-c8a22727e954/kube-rbac-proxy/0.log" Sep 10 00:03:03.584378 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:03.584351 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dmks5_59a84f25-1c0e-4a2d-b6a0-7f4b4aff51af/dns-node-resolver/0.log" Sep 10 00:03:04.058737 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:04.058698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" event={"ID":"5f1da6c5-5a69-4862-a8e5-4c91d591c5cd","Type":"ContainerStarted","Data":"866664baa475abe6ddd5caf92bc2656adec885243a874c6a78dcd4182cc18d7b"} Sep 10 00:03:04.059181 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:04.059014 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:04.079616 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:04.079560 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" podStartSLOduration=2.079522611 podStartE2EDuration="2.079522611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 00:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:03:04.078951223 +0000 UTC m=+896.286090995" watchObservedRunningTime="2026-09-10 00:03:04.079522611 +0000 UTC m=+896.286662380" Sep 10 00:03:04.109294 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:04.109244 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8648f96b6b-z5w6h_ca8f41dc-9f5a-4bb4-9396-51490c020315/registry/0.log" Sep 10 00:03:04.139598 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:04.139571 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5ksjl_44d663d3-f7dc-4270-94cc-696d3b55ea0c/node-ca/0.log" Sep 10 00:03:05.167022 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:05.166993 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-947759788-x4q2t_e78d58b9-4f18-4f8a-bf6e-7c86427779f2/kube-auth-proxy/0.log" Sep 10 00:03:05.904354 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:05.904325 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fkgvh_83f7ef84-a989-471a-ae8e-b12cad0c9279/serve-healthcheck-canary/0.log" Sep 10 00:03:06.344936 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:06.344909 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kxtp9_07f5c089-1b0e-4b58-b03b-8c1a586eaf28/kube-rbac-proxy/0.log" Sep 10 00:03:06.363553 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:06.363530 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kxtp9_07f5c089-1b0e-4b58-b03b-8c1a586eaf28/exporter/0.log" Sep 10 00:03:06.383297 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:06.383253 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kxtp9_07f5c089-1b0e-4b58-b03b-8c1a586eaf28/extractor/0.log" Sep 10 00:03:07.234059 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:07.234030 2574 scope.go:117] "RemoveContainer" containerID="20775065af0a0a0f723e33cc82a94ec7562906e5fd35bf2e2529d705b9af6614" Sep 10 00:03:07.234274 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:03:07.234251 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-4vgl4_llm(bbe87c69-fda4-4a81-8a9b-e929cfd69be6)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-4vgl4" podUID="bbe87c69-fda4-4a81-8a9b-e929cfd69be6" Sep 10 00:03:08.272268 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:08.272229 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 10 00:03:08.272737 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:08.272421 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 10 00:03:08.531718 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:08.531642 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pkslh_25d4eb97-dc5e-456f-bf97-d20ec3b6c0a4/manager/0.log" Sep 10 00:03:08.569244 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:08.569214 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jm546_fce0520e-d5e7-4e3f-9532-a1765adffbad/manager/0.log" Sep 10 00:03:08.736623 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:08.736592 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7547cdb9fd-hdptp_2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545/manager/0.log" Sep 10 00:03:08.751610 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:08.751582 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7547cdb9fd-hdptp_2fdb9a5b-c5f8-4ed9-bc4d-6acf27dda545/copy-manifests/0.log" Sep 10 00:03:09.987537 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:09.987483 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-d825g_2428e147-b977-462c-a13a-9099c45f03b0/openshift-lws-operator/0.log" Sep 10 00:03:10.077789 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:10.077763 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6zkw7/perf-node-gather-daemonset-q4kpg" Sep 10 00:03:10.234345 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:10.234317 2574 scope.go:117] "RemoveContainer" containerID="eb723aaae2e7d7d5e56188c51d784c3b8e7377ee6f28f1513492034bdd9f8a6c" Sep 10 00:03:10.234509 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:03:10.234485 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-b82mw_llm(33277331-9456-4b26-ad88-aa235abb54d1)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-b82mw" podUID="33277331-9456-4b26-ad88-aa235abb54d1" Sep 10 00:03:11.935893 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:11.935775 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-947759788-x4q2t_e78d58b9-4f18-4f8a-bf6e-7c86427779f2/kube-auth-proxy/0.log" Sep 10 00:03:13.171856 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:13.171826 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-947759788-x4q2t_e78d58b9-4f18-4f8a-bf6e-7c86427779f2/kube-auth-proxy/0.log" Sep 10 00:03:16.476830 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:16.476799 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cczv2_e884e0c1-c063-4c91-a1d2-012bc3980235/kube-multus-additional-cni-plugins/0.log" Sep 10 00:03:16.496983 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:16.496963 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cczv2_e884e0c1-c063-4c91-a1d2-012bc3980235/egress-router-binary-copy/0.log" Sep 10 00:03:16.516230 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:16.516198 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cczv2_e884e0c1-c063-4c91-a1d2-012bc3980235/cni-plugins/0.log" Sep 10 00:03:16.535399 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:16.535378 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cczv2_e884e0c1-c063-4c91-a1d2-012bc3980235/bond-cni-plugin/0.log" Sep 10 00:03:16.554686 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:16.554665 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cczv2_e884e0c1-c063-4c91-a1d2-012bc3980235/routeoverride-cni/0.log" Sep 10 00:03:16.578084 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:16.578069 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cczv2_e884e0c1-c063-4c91-a1d2-012bc3980235/whereabouts-cni-bincopy/0.log" Sep 10 00:03:16.596543 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:16.596526 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cczv2_e884e0c1-c063-4c91-a1d2-012bc3980235/whereabouts-cni/0.log" Sep 10 00:03:16.935746 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:16.935722 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xvjjv_c8b59892-f5c3-4d8f-9950-652dd1c762cb/kube-multus/0.log" Sep 10 00:03:16.995007 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:16.994982 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gmqgc_eb88e26c-b785-40be-b781-088af7d86fe2/network-metrics-daemon/0.log" Sep 10 00:03:17.011739 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.011716 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gmqgc_eb88e26c-b785-40be-b781-088af7d86fe2/kube-rbac-proxy/0.log" Sep 10 00:03:17.792160 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.792128 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-controller/0.log" Sep 10 00:03:17.807776 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.807754 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/0.log" Sep 10 00:03:17.807776 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.807769 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovn-acl-logging/1.log" Sep 10 00:03:17.823261 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.823239 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/kube-rbac-proxy-node/0.log" Sep 10 00:03:17.841861 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.841838 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 00:03:17.860978 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.860962 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/northd/0.log" Sep 10 00:03:17.879325 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.879308 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/nbdb/0.log" Sep 10 00:03:17.898209 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.898195 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/sbdb/0.log" Sep 10 00:03:17.938145 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:17.938122 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwxwt_fdda7649-1315-4edf-a2ca-31ae6d15ddbf/ovnkube-controller/0.log" Sep 10 00:03:18.237313 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:18.237222 2574 scope.go:117] "RemoveContainer" containerID="2f0e5dceef35d5edd9659f2534103ae35f85d93bfaf5dc685e503a0b0eb32221" Sep 10 00:03:18.237498 ip-10-0-129-102 kubenswrapper[2574]: E0910 00:03:18.237475 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8_llm(3214c6d5-8835-4e3f-b72b-3a37db599b7d)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-nl4c8" podUID="3214c6d5-8835-4e3f-b72b-3a37db599b7d" Sep 10 00:03:19.361047 ip-10-0-129-102 kubenswrapper[2574]: I0910 00:03:19.360970 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-6mppg_dd26812f-6704-4784-98a0-a757cc61c378/network-check-target-container/0.log"