Sep 16 19:04:50.099278 ip-10-0-139-17 systemd[1]: Starting Kubernetes Kubelet... Sep 16 19:04:50.509875 ip-10-0-139-17 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 19:04:50.509875 ip-10-0-139-17 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 19:04:50.509875 ip-10-0-139-17 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 19:04:50.509875 ip-10-0-139-17 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 19:04:50.509875 ip-10-0-139-17 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 19:04:50.512841 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.512755 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 19:04:50.519286 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519270 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519287 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519292 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519295 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519299 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519302 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519305 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519310 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519314 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519318 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519321 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519323 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519326 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:04:50.519326 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519329 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519336 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519338 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519341 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519344 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519346 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519349 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519351 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519354 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519357 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519359 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519362 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519364 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519368 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519371 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519374 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519377 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519380 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519382 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:04:50.519678 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519385 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519387 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519390 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519393 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519395 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519405 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519410 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519413 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519415 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519418 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519421 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519423 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519426 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519428 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519431 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519434 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519436 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519439 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:04:50.520271 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519441 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519444 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519446 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519449 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519451 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519454 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519457 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519460 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519462 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519465 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519467 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519470 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519472 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519475 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519477 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519480 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519483 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519485 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519488 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519490 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:04:50.520872 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519493 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519496 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519498 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519500 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519503 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519506 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519508 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519511 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519513 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519516 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519519 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519521 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519524 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519527 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519530 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519533 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519535 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519538 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519541 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519543 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:04:50.521653 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519546 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519549 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519551 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519553 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519556 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519558 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519561 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519563 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.519566 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521848 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521863 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521869 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521874 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521879 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521884 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521889 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521893 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521898 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521902 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521906 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:04:50.522523 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521910 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521914 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521918 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521923 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521927 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521931 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521935 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521939 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521944 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521948 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521952 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521956 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521960 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521964 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521968 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521973 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521980 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.521986 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522013 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522018 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:04:50.523396 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522023 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522027 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522035 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522040 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522044 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522049 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522053 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522057 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522061 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522066 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522070 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522074 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522078 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522082 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522087 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522091 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522097 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522102 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522107 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:04:50.524355 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522111 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522115 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522120 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522124 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522128 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522133 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522137 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522141 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522145 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522150 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522155 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522158 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522162 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522166 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522172 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522176 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522181 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522185 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522190 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522194 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:04:50.525151 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522197 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522202 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522206 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522210 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522217 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522222 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522227 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522232 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522236 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522240 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522245 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522249 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522254 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522260 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522265 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522270 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522273 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522277 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522282 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:04:50.525649 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522287 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522291 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522295 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522299 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522303 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522308 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522312 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522318 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522322 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522326 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522906 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522914 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522919 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522924 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522928 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522933 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522937 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522941 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522945 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522950 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522954 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:04:50.526220 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522959 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522962 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522967 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522971 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522976 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522981 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.522986 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523009 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523014 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523018 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523022 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523026 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523030 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523034 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523038 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523043 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523047 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523051 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523057 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:04:50.526766 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523061 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523065 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523070 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523075 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523080 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523084 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523088 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523092 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523097 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523101 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523108 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523114 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523120 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523125 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523129 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523134 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523138 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523156 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523161 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523166 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:04:50.527267 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523171 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523175 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523179 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523183 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523187 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523192 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523196 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523200 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523205 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523209 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523213 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523219 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523223 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523227 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523232 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523239 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523245 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523249 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523257 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:04:50.527964 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523263 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523267 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523272 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523276 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523280 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523285 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523289 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523294 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523298 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523302 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523307 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523310 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523315 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523320 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523324 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523328 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523333 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523337 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523341 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523346 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:04:50.528466 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523350 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523354 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523358 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523363 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523368 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523373 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523377 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523381 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523385 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.523389 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523497 2562 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523549 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523560 2562 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523566 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523572 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523578 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523584 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523591 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523595 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523600 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523606 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 19:04:50.528973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523611 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523616 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523621 2562 flags.go:64] FLAG: --cgroup-root="" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523626 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523631 2562 flags.go:64] FLAG: --client-ca-file="" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523637 2562 flags.go:64] FLAG: --cloud-provider="external" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523642 2562 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523648 2562 flags.go:64] FLAG: --cluster-domain="" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523652 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523657 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523662 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523668 2562 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523674 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523679 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523684 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523693 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523698 2562 flags.go:64] FLAG: --contention-profiling="false" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523702 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523707 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523712 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523717 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523723 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523728 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523733 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523738 2562 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 19:04:50.529606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523742 2562 flags.go:64] FLAG: --enable-server="true" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523747 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523755 2562 flags.go:64] FLAG: --event-burst="100" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523760 2562 flags.go:64] FLAG: --event-qps="50" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523764 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523770 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523775 2562 flags.go:64] FLAG: --eviction-hard="" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523781 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523786 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523791 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523796 2562 flags.go:64] FLAG: --eviction-soft="" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523801 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523805 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523811 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523816 2562 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523820 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523825 2562 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523829 2562 flags.go:64] FLAG: --feature-gates="" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523836 2562 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523842 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523847 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523852 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523859 2562 flags.go:64] FLAG: --healthz-port="10248" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523864 2562 flags.go:64] FLAG: --help="false" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523868 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.530279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523874 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523879 2562 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523884 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523889 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523894 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523899 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523904 2562 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523908 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523913 2562 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523918 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523923 2562 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523928 2562 flags.go:64] FLAG: --kube-reserved="" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523933 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523937 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523942 2562 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523947 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523951 2562 flags.go:64] FLAG: --lock-file="" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523956 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523961 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523966 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523975 2562 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523980 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.523985 2562 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 19:04:50.530956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524006 2562 flags.go:64] FLAG: --logging-format="text" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524011 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524017 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524022 2562 flags.go:64] FLAG: --manifest-url="" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524026 2562 flags.go:64] FLAG: --manifest-url-header="" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524033 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524041 2562 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524047 2562 flags.go:64] FLAG: --max-pods="110" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524052 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524057 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524062 2562 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524066 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524071 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524076 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524080 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524091 2562 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524096 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524101 2562 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524106 2562 flags.go:64] FLAG: --pod-cidr="" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524110 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524119 2562 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524124 2562 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524129 2562 flags.go:64] FLAG: --pods-per-core="0" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524134 2562 flags.go:64] FLAG: --port="10250" Sep 16 19:04:50.531580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524139 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524143 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07719835d29cebfc8" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524148 2562 flags.go:64] FLAG: --qos-reserved="" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524154 2562 flags.go:64] FLAG: --read-only-port="10255" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524159 2562 flags.go:64] FLAG: --register-node="true" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524163 2562 flags.go:64] FLAG: --register-with-taints="" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524169 2562 flags.go:64] FLAG: --registry-burst="10" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524175 2562 flags.go:64] FLAG: --registry-qps="5" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524179 2562 flags.go:64] FLAG: --reserved-cpus="" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524183 2562 flags.go:64] FLAG: --reserved-memory="" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524189 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524194 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524199 2562 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524204 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524211 2562 flags.go:64] FLAG: --runonce="false" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524218 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524223 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524228 2562 flags.go:64] FLAG: --seccomp-default="false" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524233 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524238 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524243 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524248 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524253 2562 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524257 2562 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524262 2562 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 19:04:50.532268 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524267 2562 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524271 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524276 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524281 2562 flags.go:64] FLAG: --system-cgroups="" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524286 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524294 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524299 2562 flags.go:64] FLAG: --tls-cert-file="" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524304 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524310 2562 flags.go:64] FLAG: --tls-min-version="" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524314 2562 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524319 2562 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524324 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524329 2562 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524334 2562 flags.go:64] FLAG: --v="2" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524340 2562 flags.go:64] FLAG: --version="false" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524346 2562 flags.go:64] FLAG: --vmodule="" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524352 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.524357 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524543 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524550 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524556 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524561 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524566 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:04:50.532887 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524570 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524576 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524580 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524584 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524588 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524593 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524598 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524602 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524606 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524610 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524615 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524619 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524623 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524627 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524631 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524635 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524639 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524643 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524647 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524652 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:04:50.533477 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524656 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524660 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524664 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524668 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524696 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524704 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524709 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524714 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524719 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524723 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524728 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524732 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524737 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524741 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524745 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524749 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524754 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524758 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524762 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524766 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:04:50.534011 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524770 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524774 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524778 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524783 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524787 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524791 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524795 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524800 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524806 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524812 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524817 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524822 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524826 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524831 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524836 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524841 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524845 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524849 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524855 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:04:50.534591 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524860 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524864 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524868 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524873 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524877 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524881 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524885 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524889 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524893 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524897 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524901 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524905 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524909 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524913 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524918 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524922 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524925 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524929 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524934 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524938 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:04:50.535078 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524942 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524946 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524950 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524954 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524958 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524965 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524970 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524975 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524981 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.524986 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.525007 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.525012 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.525018 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.525023 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:04:50.535575 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.525027 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.525744 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.533562 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.533577 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.534528 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.534536 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534592 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534598 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534602 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534605 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534608 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534611 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534614 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534616 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534619 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:04:50.535918 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534621 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534624 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534627 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534629 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534632 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534635 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534637 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534640 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534643 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534645 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534648 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534651 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534653 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534656 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534660 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534664 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534667 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534670 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534672 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534675 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:04:50.536360 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534679 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534682 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534685 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534687 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534690 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534692 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534695 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534698 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534700 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534703 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534706 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534708 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534711 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534713 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534717 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534721 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534724 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534727 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534730 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534733 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:04:50.536838 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534736 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534738 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534741 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534744 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534747 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534750 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534753 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534755 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534758 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534760 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534763 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534766 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534769 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534771 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534774 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534777 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534779 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534782 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534784 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:04:50.537383 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534787 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534789 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534792 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534794 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534797 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534799 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534802 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534805 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534807 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534809 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534813 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534815 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534818 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534820 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534822 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534827 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534831 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534834 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534837 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534840 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:04:50.537844 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534843 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534846 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534849 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534852 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534855 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534858 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534861 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534864 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534867 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534870 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534873 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.534878 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.534987 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535009 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:04:50.538354 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535012 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535015 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535018 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535021 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535024 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535027 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535030 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535032 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535035 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535038 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535041 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535044 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535046 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535049 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535052 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535055 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535057 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535060 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535063 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535065 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:04:50.538717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535068 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535071 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535073 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535076 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535079 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535082 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535085 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535088 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535091 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535093 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535096 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535098 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535101 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535103 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535106 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535109 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535111 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535114 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535116 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535119 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:04:50.539227 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535122 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535126 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535129 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535131 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535134 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535138 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535142 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535144 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535148 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535151 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535154 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535157 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535160 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535162 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535165 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535168 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535171 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535175 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535178 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:04:50.539717 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535185 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535188 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535191 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535194 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535196 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535199 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535201 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535204 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535207 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535209 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535212 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535214 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535217 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535219 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535222 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535224 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535227 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535229 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535232 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:04:50.540185 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535235 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535237 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535240 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535242 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535245 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535247 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535249 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535252 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535255 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535257 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535260 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535262 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535265 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535268 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535271 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535274 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535276 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535279 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:04:50.540655 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:50.535281 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:04:50.541111 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.535286 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 19:04:50.541111 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.535381 2562 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 19:04:50.541111 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.537785 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 19:04:50.541111 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.538873 2562 server.go:1019] "Starting client certificate rotation" Sep 16 19:04:50.541111 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.538965 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 19:04:50.541111 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.539011 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 19:04:50.541770 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.541757 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 19:04:50.543300 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.543282 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 19:04:50.558821 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.558796 2562 log.go:25] "Validated CRI v1 runtime API" Sep 16 19:04:50.562538 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.562522 2562 log.go:25] "Validated CRI v1 image API" Sep 16 19:04:50.563348 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.563333 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 19:04:50.566731 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.566714 2562 fs.go:135] Filesystem UUIDs: map[74307176-bfe9-423d-8e0b-65614d22b2a8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a65cb6d3-fedc-4900-9e09-d3429197b882:/dev/nvme0n1p3] Sep 16 19:04:50.566798 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.566731 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 19:04:50.572139 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.572027 2562 manager.go:216] Machine: {Timestamp:2026-09-16 19:04:50.571485725 +0000 UTC m=+0.369562531 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101729 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c10c40cd6b8a3f9fd5365a3538b39 SystemUUID:ec2c10c4-0cd6-b8a3-f9fd-5365a3538b39 BootID:299810cf-6c41-4fb0-88f4-7ac04f447d10 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:e9:f2:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:e9:f2:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:3c:1d:8d:1d:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 19:04:50.572139 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.572131 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 19:04:50.572270 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.572209 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 19:04:50.574509 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.574493 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 19:04:50.574816 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.574794 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 19:04:50.574968 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.574821 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-17.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 19:04:50.576345 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.576334 2562 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 19:04:50.576389 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.576348 2562 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 19:04:50.576389 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.576361 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 19:04:50.577369 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.577358 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 19:04:50.578614 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.578603 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 19:04:50.584194 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.584182 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 16 19:04:50.584309 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.584299 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 19:04:50.584337 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.584328 2562 kubelet.go:486] "Attempting to sync node with API server" Sep 16 19:04:50.584365 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.584338 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 19:04:50.584365 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.584354 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 19:04:50.584365 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.584363 2562 kubelet.go:398] "Adding apiserver pod source" Sep 16 19:04:50.584470 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.584372 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 19:04:50.584729 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.584714 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 19:04:50.584784 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.584737 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 19:04:50.587091 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.587074 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 19:04:50.590035 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.590021 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 19:04:50.590095 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.590056 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 19:04:50.592097 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592086 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 19:04:50.592145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592106 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 19:04:50.592145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592116 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 19:04:50.592145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592123 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 19:04:50.592145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592132 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 19:04:50.592145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592138 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 19:04:50.592272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592151 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 19:04:50.592272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592157 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 19:04:50.592272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592163 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 19:04:50.592272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592169 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 19:04:50.592272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592183 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 19:04:50.592272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.592194 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 19:04:50.593049 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.593040 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 19:04:50.593049 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.593050 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 19:04:50.594235 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.594207 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 19:04:50.594235 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.594209 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 19:04:50.595765 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.595748 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-17.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 19:04:50.596805 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.596791 2562 server.go:1268] "Started kubelet" Sep 16 19:04:50.596888 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.596856 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 19:04:50.596978 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.596949 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 19:04:50.597027 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.597018 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 19:04:50.597699 ip-10-0-139-17 systemd[1]: Started Kubernetes Kubelet. Sep 16 19:04:50.598610 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.598595 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 19:04:50.605567 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.605526 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 19:04:50.605803 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.605782 2562 server.go:310] "Adding debug handlers to kubelet server" Sep 16 19:04:50.608976 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.607841 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-17.ec2.internal.18d5e285a636a874 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-17.ec2.internal,UID:ip-10-0-139-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-17.ec2.internal,},FirstTimestamp:2026-09-16 19:04:50.59676786 +0000 UTC m=+0.394844665,LastTimestamp:2026-09-16 19:04:50.59676786 +0000 UTC m=+0.394844665,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-17.ec2.internal,}" Sep 16 19:04:50.611570 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.611551 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 19:04:50.612308 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.612293 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 19:04:50.612429 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.612410 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:50.613041 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613022 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 19:04:50.613135 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613042 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 19:04:50.613135 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613025 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 19:04:50.613135 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613119 2562 factory.go:55] Registering systemd factory Sep 16 19:04:50.613135 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613130 2562 factory.go:223] Registration of the systemd container factory successfully Sep 16 19:04:50.613279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613172 2562 reconstruct.go:97] "Volume reconstruction finished" Sep 16 19:04:50.613279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613180 2562 reconciler.go:29] "Reconciler: start to sync state" Sep 16 19:04:50.613337 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613314 2562 factory.go:153] Registering CRI-O factory Sep 16 19:04:50.613337 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613323 2562 factory.go:223] Registration of the crio container factory successfully Sep 16 19:04:50.613395 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613372 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 19:04:50.613395 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613393 2562 factory.go:103] Registering Raw factory Sep 16 19:04:50.613451 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613402 2562 manager.go:1196] Started watching for new ooms in manager Sep 16 19:04:50.613792 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.613778 2562 manager.go:349] Starting recovery of all containers Sep 16 19:04:50.618985 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.618957 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 19:04:50.619086 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.619038 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 19:04:50.624253 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.624212 2562 manager.go:354] Recovery completed Sep 16 19:04:50.629867 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.629853 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:04:50.632023 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.632005 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:04:50.632092 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.632040 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:04:50.632092 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.632054 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:04:50.632658 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.632647 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 19:04:50.632658 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.632658 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 19:04:50.632761 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.632673 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 16 19:04:50.632761 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.632650 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:04:50.634002 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.633925 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-17.ec2.internal.18d5e285a8509f41 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-17.ec2.internal,UID:ip-10-0-139-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-17.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-17.ec2.internal,},FirstTimestamp:2026-09-16 19:04:50.632023873 +0000 UTC m=+0.430100689,LastTimestamp:2026-09-16 19:04:50.632023873 +0000 UTC m=+0.430100689,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-17.ec2.internal,}" Sep 16 19:04:50.635029 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.635013 2562 policy_none.go:49] "None policy: Start" Sep 16 19:04:50.635126 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.635043 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 19:04:50.635126 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.635058 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 19:04:50.635126 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.635011 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:04:50.635126 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.635126 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:04:50.635307 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.635141 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:04:50.637201 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.637187 2562 policy_none.go:47] "Start" Sep 16 19:04:50.638333 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.638318 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xrlcn" Sep 16 19:04:50.647198 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.647158 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xrlcn" Sep 16 19:04:50.648030 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.647942 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-17.ec2.internal.18d5e285a850fa8c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-17.ec2.internal,UID:ip-10-0-139-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-17.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-17.ec2.internal,},FirstTimestamp:2026-09-16 19:04:50.632047244 +0000 UTC m=+0.430124052,LastTimestamp:2026-09-16 19:04:50.632047244 +0000 UTC m=+0.430124052,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-17.ec2.internal,}" Sep 16 19:04:50.672701 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.672680 2562 manager.go:337] "Starting Device Plugin manager" Sep 16 19:04:50.690272 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.672725 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 19:04:50.690272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.672739 2562 server.go:87] "Starting device plugin registration server" Sep 16 19:04:50.690272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.673070 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 19:04:50.690272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.673082 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 19:04:50.690272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.673188 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 19:04:50.690272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.673278 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 19:04:50.690272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.673286 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 19:04:50.690272 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.673726 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 19:04:50.690272 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.673777 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:50.706505 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.706479 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 19:04:50.707688 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.707664 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 19:04:50.707795 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.707727 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 19:04:50.707795 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.707752 2562 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 19:04:50.707890 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.707798 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 19:04:50.710133 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.710113 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:04:50.773835 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.773751 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:04:50.775712 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.775694 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:04:50.775787 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.775727 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:04:50.775787 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.775740 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:04:50.775787 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.775765 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.785520 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.785504 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.785577 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.785531 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-17.ec2.internal\": node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:50.801887 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.801867 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:50.807878 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.807861 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal"] Sep 16 19:04:50.807934 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.807925 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:04:50.809347 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.809331 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:04:50.809398 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.809360 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:04:50.809398 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.809370 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:04:50.811704 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.811692 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:04:50.811818 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.811803 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.811861 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.811836 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:04:50.812522 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.812505 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:04:50.812594 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.812507 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:04:50.812594 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.812553 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:04:50.812594 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.812565 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:04:50.812594 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.812529 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:04:50.812706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.812607 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:04:50.814365 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.814350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06f029b86425ea0c165607a9c36ed11c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal\" (UID: \"06f029b86425ea0c165607a9c36ed11c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.814416 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.814374 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06f029b86425ea0c165607a9c36ed11c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal\" (UID: \"06f029b86425ea0c165607a9c36ed11c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.814416 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.814396 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/322d0d38fd19fa48489dd88803493fc9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-17.ec2.internal\" (UID: \"322d0d38fd19fa48489dd88803493fc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.814886 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.814873 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.814922 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.814897 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:04:50.815656 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.815642 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:04:50.815711 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.815665 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:04:50.815711 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.815688 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:04:50.843610 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.843589 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-17.ec2.internal\" not found" node="ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.848117 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.848102 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-17.ec2.internal\" not found" node="ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.902267 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:50.902241 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:50.915389 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.915362 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06f029b86425ea0c165607a9c36ed11c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal\" (UID: \"06f029b86425ea0c165607a9c36ed11c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.915482 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.915393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06f029b86425ea0c165607a9c36ed11c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal\" (UID: \"06f029b86425ea0c165607a9c36ed11c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.915482 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.915410 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/322d0d38fd19fa48489dd88803493fc9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-17.ec2.internal\" (UID: \"322d0d38fd19fa48489dd88803493fc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.915482 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.915449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/322d0d38fd19fa48489dd88803493fc9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-17.ec2.internal\" (UID: \"322d0d38fd19fa48489dd88803493fc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.915482 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.915471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06f029b86425ea0c165607a9c36ed11c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal\" (UID: \"06f029b86425ea0c165607a9c36ed11c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" Sep 16 19:04:50.915617 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:50.915471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06f029b86425ea0c165607a9c36ed11c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal\" (UID: \"06f029b86425ea0c165607a9c36ed11c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" Sep 16 19:04:51.002549 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:51.002516 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:51.103412 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:51.103341 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:51.146829 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:51.146798 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" Sep 16 19:04:51.150760 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:51.150735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal" Sep 16 19:04:52.452658 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:52.452628 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:52.553237 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:52.553192 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:52.653745 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:52.653673 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:52.754375 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:52.754346 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:52.787824 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:52.787793 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 19:04:52.788249 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:52.787939 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 19:04:52.854471 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:52.854442 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:52.902629 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:52.902593 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:59:51 +0000 UTC" deadline="2028-04-23 22:16:00.259631841 +0000 UTC" Sep 16 19:04:52.902765 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:52.902637 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14043h11m7.357010938s" Sep 16 19:04:52.954958 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:52.954925 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:53.005533 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.005506 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:04:53.006668 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.006652 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:04:53.006740 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.006685 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:04:53.006740 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.006694 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:04:53.056116 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.056083 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:53.082334 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.082309 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:04:53.103756 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:53.103725 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f029b86425ea0c165607a9c36ed11c.slice/crio-74c00570737dd02b51bbb6f82e04231fb03487fcde3a3bca573f0421fb52e5d6 WatchSource:0}: Error finding container 74c00570737dd02b51bbb6f82e04231fb03487fcde3a3bca573f0421fb52e5d6: Status 404 returned error can't find the container with id 74c00570737dd02b51bbb6f82e04231fb03487fcde3a3bca573f0421fb52e5d6 Sep 16 19:04:53.103928 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:53.103913 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322d0d38fd19fa48489dd88803493fc9.slice/crio-13013d67f6bf777474c1fe33d582967ddd009c4f52ae8084afe29cb0402efedc WatchSource:0}: Error finding container 13013d67f6bf777474c1fe33d582967ddd009c4f52ae8084afe29cb0402efedc: Status 404 returned error can't find the container with id 13013d67f6bf777474c1fe33d582967ddd009c4f52ae8084afe29cb0402efedc Sep 16 19:04:53.108505 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.108483 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:04:53.156783 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.156746 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:53.257383 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.257311 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:53.358282 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.358246 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-17.ec2.internal\" not found" Sep 16 19:04:53.359064 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.359049 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:04:53.362048 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.362031 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" Sep 16 19:04:53.366329 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.366313 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:04:53.375558 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.375542 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 19:04:53.376437 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.376426 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal" Sep 16 19:04:53.385524 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.385509 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 19:04:53.834541 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.834505 2562 apiserver.go:52] "Watching apiserver" Sep 16 19:04:53.844471 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.844440 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 19:04:53.845349 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.845326 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hjmsb","kube-system/konnectivity-agent-pjtv9","kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal","openshift-image-registry/node-ca-ft7jx","openshift-multus/multus-additional-cni-plugins-r2hzb","openshift-multus/network-metrics-daemon-mzvlv","openshift-network-operator/iptables-alerter-vjhwb","openshift-ovn-kubernetes/ovnkube-node-w7hqb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k","openshift-cluster-node-tuning-operator/tuned-glj5r","openshift-dns/node-resolver-vq45m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal","openshift-multus/multus-5tk6q","openshift-network-diagnostics/network-check-target-cnrd5"] Sep 16 19:04:53.849189 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.849171 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.851194 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.851170 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:04:53.851292 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.851221 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.852047 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.852033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wmx5s\"" Sep 16 19:04:53.852122 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.852108 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 19:04:53.852359 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.852347 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 19:04:53.852984 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.852968 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 19:04:53.853933 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.853917 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:04:53.854023 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.853994 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:04:53.854494 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.854469 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 19:04:53.854824 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.854699 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 19:04:53.854824 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.854784 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 19:04:53.855226 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.855208 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jjnnw\"" Sep 16 19:04:53.857208 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.857189 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 19:04:53.857424 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.857407 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.859613 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.859594 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.861310 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.861284 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 19:04:53.861949 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.861930 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:53.862055 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.862025 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:04:53.862845 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.862826 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 19:04:53.862940 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.862853 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 19:04:53.863343 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.863124 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 19:04:53.863343 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.863161 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 19:04:53.863343 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.863208 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 19:04:53.863681 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.863472 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n7kf9\"" Sep 16 19:04:53.863681 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.863560 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-96mf5\"" Sep 16 19:04:53.864519 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.864282 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.867195 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.867175 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 19:04:53.867492 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.867472 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:53.867579 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.867534 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:04:53.867687 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.867671 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 19:04:53.867772 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.867723 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r7twt\"" Sep 16 19:04:53.869695 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.869672 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:04:53.870119 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.869982 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 19:04:53.872111 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.872091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.872982 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.872963 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fxsz9\"" Sep 16 19:04:53.873251 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.873232 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 19:04:53.873251 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.873245 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 19:04:53.874315 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874295 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:53.874405 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874328 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5ct6\" (UniqueName: \"kubernetes.io/projected/f01f26bd-0336-4941-bc54-62734d65f013-kube-api-access-s5ct6\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.874405 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874356 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjpw8\" (UniqueName: \"kubernetes.io/projected/9f54592e-2056-4dd4-969c-eaeec3227210-kube-api-access-gjpw8\") pod \"node-ca-ft7jx\" (UID: \"9f54592e-2056-4dd4-969c-eaeec3227210\") " pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.874405 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f01f26bd-0336-4941-bc54-62734d65f013-system-cni-dir\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.874559 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac8a58c4-152a-421d-8cf6-609cb576d716-host-slash\") pod \"iptables-alerter-vjhwb\" (UID: \"ac8a58c4-152a-421d-8cf6-609cb576d716\") " pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.874559 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghqb\" (UniqueName: \"kubernetes.io/projected/ac8a58c4-152a-421d-8cf6-609cb576d716-kube-api-access-sghqb\") pod \"iptables-alerter-vjhwb\" (UID: \"ac8a58c4-152a-421d-8cf6-609cb576d716\") " pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.874559 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-var-lib-kubelet\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.874559 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f54592e-2056-4dd4-969c-eaeec3227210-host\") pod \"node-ca-ft7jx\" (UID: \"9f54592e-2056-4dd4-969c-eaeec3227210\") " pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.874559 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874511 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-var-lib-cni-multus\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.874559 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:53.874559 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-run-multus-certs\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.874884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f01f26bd-0336-4941-bc54-62734d65f013-cnibin\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.874884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-device-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.874884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-run-k8s-cni-cncf-io\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.874884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874668 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqr79\" (UniqueName: \"kubernetes.io/projected/10b511a5-508f-4ce7-988a-7abc0d8e70da-kube-api-access-xqr79\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:53.874884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f66049e9-573e-4aa1-add4-b930be25fb66-multus-daemon-config\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.874884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcc5m\" (UniqueName: \"kubernetes.io/projected/f66049e9-573e-4aa1-add4-b930be25fb66-kube-api-access-pcc5m\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.876556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.875699 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 19:04:53.876556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.875866 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 19:04:53.876556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.875889 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 19:04:53.876556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.874761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftvc\" (UniqueName: \"kubernetes.io/projected/6466fa2e-35f0-499b-a564-a878278d3377-kube-api-access-bftvc\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.876556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876277 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9t45s\"" Sep 16 19:04:53.876556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:04:53.876556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876434 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac8a58c4-152a-421d-8cf6-609cb576d716-iptables-alerter-script\") pod \"iptables-alerter-vjhwb\" (UID: \"ac8a58c4-152a-421d-8cf6-609cb576d716\") " pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.876556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-run-netns\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.876556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876491 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f01f26bd-0336-4941-bc54-62734d65f013-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.877058 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9f54592e-2056-4dd4-969c-eaeec3227210-serviceca\") pod \"node-ca-ft7jx\" (UID: \"9f54592e-2056-4dd4-969c-eaeec3227210\") " pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.877058 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-etc-kubernetes\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.877058 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876675 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f66049e9-573e-4aa1-add4-b930be25fb66-cni-binary-copy\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.877058 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876701 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-hostroot\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.877058 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.877058 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.877058 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-sys-fs\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.877058 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876875 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-system-cni-dir\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.877058 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.876922 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-multus-cni-dir\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877080 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6h2vz\"" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-multus-socket-dir-parent\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877137 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-var-lib-cni-bin\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877169 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-multus-conf-dir\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877200 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f01f26bd-0336-4941-bc54-62734d65f013-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877232 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f01f26bd-0336-4941-bc54-62734d65f013-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877287 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-registration-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-cnibin\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877344 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f01f26bd-0336-4941-bc54-62734d65f013-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.877517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877372 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-socket-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.877947 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.877909 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 19:04:53.878919 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.878899 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 19:04:53.879499 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.879470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.881805 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.881786 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vwlzg\"" Sep 16 19:04:53.882157 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.882112 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 19:04:53.882157 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.882131 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 19:04:53.882157 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.882147 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 19:04:53.882855 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.882835 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 19:04:53.883028 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.882885 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 19:04:53.883114 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.883068 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 19:04:53.910586 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.910561 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jgr5l" Sep 16 19:04:53.917917 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.917898 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jgr5l" Sep 16 19:04:53.962976 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.962951 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 19:04:53.963865 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.963818 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal" event={"ID":"322d0d38fd19fa48489dd88803493fc9","Type":"ContainerStarted","Data":"13013d67f6bf777474c1fe33d582967ddd009c4f52ae8084afe29cb0402efedc"} Sep 16 19:04:53.965321 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.965295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" event={"ID":"06f029b86425ea0c165607a9c36ed11c","Type":"ContainerStarted","Data":"74c00570737dd02b51bbb6f82e04231fb03487fcde3a3bca573f0421fb52e5d6"} Sep 16 19:04:53.977667 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-run-k8s-cni-cncf-io\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.977796 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqr79\" (UniqueName: \"kubernetes.io/projected/10b511a5-508f-4ce7-988a-7abc0d8e70da-kube-api-access-xqr79\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:53.977796 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977701 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f66049e9-573e-4aa1-add4-b930be25fb66-multus-daemon-config\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.977796 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcc5m\" (UniqueName: \"kubernetes.io/projected/f66049e9-573e-4aa1-add4-b930be25fb66-kube-api-access-pcc5m\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.977796 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-sysctl-conf\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-lib-modules\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977853 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-run-systemd\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-run-netns\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f01f26bd-0336-4941-bc54-62734d65f013-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977907 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-ovnkube-script-lib\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977955 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqq4j\" (UniqueName: \"kubernetes.io/projected/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-kube-api-access-jqq4j\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.977979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ffe5e8fe-18fc-43f5-8f36-23b3e89025fa-konnectivity-ca\") pod \"konnectivity-agent-pjtv9\" (UID: \"ffe5e8fe-18fc-43f5-8f36-23b3e89025fa\") " pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-sysctl-d\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978024 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-run-k8s-cni-cncf-io\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-sys\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.978167 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-etc-kubernetes\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978178 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f66049e9-573e-4aa1-add4-b930be25fb66-cni-binary-copy\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978194 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-etc-kubernetes\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978228 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-run-netns\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978317 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f01f26bd-0336-4941-bc54-62734d65f013-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978478 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a31ef62-788d-40d4-b735-08587d7e341d-kubelet-config\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-system-cni-dir\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-multus-conf-dir\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f01f26bd-0336-4941-bc54-62734d65f013-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978829 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-registration-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f66049e9-573e-4aa1-add4-b930be25fb66-multus-daemon-config\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.978885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a31ef62-788d-40d4-b735-08587d7e341d-dbus\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-cnibin\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978939 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-socket-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978946 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f66049e9-573e-4aa1-add4-b930be25fb66-cni-binary-copy\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978965 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-systemd-units\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.978996 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-run-ovn\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5bf475c5-b68d-4235-b53e-ff2420b1e556-hosts-file\") pod \"node-resolver-vq45m\" (UID: \"5bf475c5-b68d-4235-b53e-ff2420b1e556\") " pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpvc\" (UniqueName: \"kubernetes.io/projected/5bf475c5-b68d-4235-b53e-ff2420b1e556-kube-api-access-mfpvc\") pod \"node-resolver-vq45m\" (UID: \"5bf475c5-b68d-4235-b53e-ff2420b1e556\") " pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac8a58c4-152a-421d-8cf6-609cb576d716-host-slash\") pod \"iptables-alerter-vjhwb\" (UID: \"ac8a58c4-152a-421d-8cf6-609cb576d716\") " pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghqb\" (UniqueName: \"kubernetes.io/projected/ac8a58c4-152a-421d-8cf6-609cb576d716-kube-api-access-sghqb\") pod \"iptables-alerter-vjhwb\" (UID: \"ac8a58c4-152a-421d-8cf6-609cb576d716\") " pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f54592e-2056-4dd4-969c-eaeec3227210-host\") pod \"node-ca-ft7jx\" (UID: \"9f54592e-2056-4dd4-969c-eaeec3227210\") " pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-tuned\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.979489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979476 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8r5\" (UniqueName: \"kubernetes.io/projected/9dff5577-ae04-4e24-9c72-f2175eb61ff8-kube-api-access-tg8r5\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-run-netns\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979524 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-run-multus-certs\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-device-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-run\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f01f26bd-0336-4941-bc54-62734d65f013-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-kubelet\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-etc-openvswitch\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979640 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-log-socket\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-cni-bin\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979060 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-multus-conf-dir\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979049 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-system-cni-dir\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979111 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-socket-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979000 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-cnibin\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979682 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-cni-netd\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979089 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-registration-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.979836 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:04:53.980074 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979896 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac8a58c4-152a-421d-8cf6-609cb576d716-host-slash\") pod \"iptables-alerter-vjhwb\" (UID: \"ac8a58c4-152a-421d-8cf6-609cb576d716\") " pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979921 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-device-dir\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftvc\" (UniqueName: \"kubernetes.io/projected/6466fa2e-35f0-499b-a564-a878278d3377-kube-api-access-bftvc\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979951 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f54592e-2056-4dd4-969c-eaeec3227210-host\") pod \"node-ca-ft7jx\" (UID: \"9f54592e-2056-4dd4-969c-eaeec3227210\") " pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dff5577-ae04-4e24-9c72-f2175eb61ff8-tmp\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979988 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-node-log\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.979992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-run-multus-certs\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-ovn-node-metrics-cert\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.980073 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs podName:10b511a5-508f-4ce7-988a-7abc0d8e70da nodeName:}" failed. No retries permitted until 2026-09-16 19:04:54.480051809 +0000 UTC m=+3.029109073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs") pod "network-metrics-daemon-mzvlv" (UID: "10b511a5-508f-4ce7-988a-7abc0d8e70da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ffe5e8fe-18fc-43f5-8f36-23b3e89025fa-agent-certs\") pod \"konnectivity-agent-pjtv9\" (UID: \"ffe5e8fe-18fc-43f5-8f36-23b3e89025fa\") " pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac8a58c4-152a-421d-8cf6-609cb576d716-iptables-alerter-script\") pod \"iptables-alerter-vjhwb\" (UID: \"ac8a58c4-152a-421d-8cf6-609cb576d716\") " pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9f54592e-2056-4dd4-969c-eaeec3227210-serviceca\") pod \"node-ca-ft7jx\" (UID: \"9f54592e-2056-4dd4-969c-eaeec3227210\") " pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-modprobe-d\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-env-overrides\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-hostroot\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-sys-fs\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.980872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-hostroot\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980336 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-ovnkube-config\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980339 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6466fa2e-35f0-499b-a564-a878278d3377-sys-fs\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf475c5-b68d-4235-b53e-ff2420b1e556-tmp-dir\") pod \"node-resolver-vq45m\" (UID: \"5bf475c5-b68d-4235-b53e-ff2420b1e556\") " pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-multus-cni-dir\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-multus-socket-dir-parent\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980425 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-var-lib-cni-bin\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980450 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f01f26bd-0336-4941-bc54-62734d65f013-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980486 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-var-lib-openvswitch\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f01f26bd-0336-4941-bc54-62734d65f013-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-systemd\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980540 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-var-lib-cni-bin\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-var-lib-kubelet\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-multus-cni-dir\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980542 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-multus-socket-dir-parent\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9f54592e-2056-4dd4-969c-eaeec3227210-serviceca\") pod \"node-ca-ft7jx\" (UID: \"9f54592e-2056-4dd4-969c-eaeec3227210\") " pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.981702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-run-openvswitch\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5ct6\" (UniqueName: \"kubernetes.io/projected/f01f26bd-0336-4941-bc54-62734d65f013-kube-api-access-s5ct6\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjpw8\" (UniqueName: \"kubernetes.io/projected/9f54592e-2056-4dd4-969c-eaeec3227210-kube-api-access-gjpw8\") pod \"node-ca-ft7jx\" (UID: \"9f54592e-2056-4dd4-969c-eaeec3227210\") " pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980713 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac8a58c4-152a-421d-8cf6-609cb576d716-iptables-alerter-script\") pod \"iptables-alerter-vjhwb\" (UID: \"ac8a58c4-152a-421d-8cf6-609cb576d716\") " pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980721 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-kubernetes\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f01f26bd-0336-4941-bc54-62734d65f013-system-cni-dir\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-var-lib-kubelet\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980823 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f01f26bd-0336-4941-bc54-62734d65f013-system-cni-dir\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-sysconfig\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-host\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980866 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-var-lib-kubelet\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-slash\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-var-lib-cni-multus\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f01f26bd-0336-4941-bc54-62734d65f013-cnibin\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980969 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f66049e9-573e-4aa1-add4-b930be25fb66-host-var-lib-cni-multus\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.980984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f01f26bd-0336-4941-bc54-62734d65f013-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.981043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f01f26bd-0336-4941-bc54-62734d65f013-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.982527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.981062 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f01f26bd-0336-4941-bc54-62734d65f013-cnibin\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:53.985615 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.985408 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 19:04:53.985615 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.985504 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:04:53.985615 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.985520 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:04:53.985615 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.985528 2562 projected.go:196] Error preparing data for projected volume kube-api-access-dpjx8 for pod openshift-network-diagnostics/network-check-target-cnrd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:04:53.985615 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:53.985574 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8 podName:49609a4b-5c7b-412e-b0ae-01d78c0ea546 nodeName:}" failed. No retries permitted until 2026-09-16 19:04:54.485561787 +0000 UTC m=+3.034619039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dpjx8" (UniqueName: "kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8") pod "network-check-target-cnrd5" (UID: "49609a4b-5c7b-412e-b0ae-01d78c0ea546") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:04:53.989851 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.989760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcc5m\" (UniqueName: \"kubernetes.io/projected/f66049e9-573e-4aa1-add4-b930be25fb66-kube-api-access-pcc5m\") pod \"multus-5tk6q\" (UID: \"f66049e9-573e-4aa1-add4-b930be25fb66\") " pod="openshift-multus/multus-5tk6q" Sep 16 19:04:53.989851 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.989785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqr79\" (UniqueName: \"kubernetes.io/projected/10b511a5-508f-4ce7-988a-7abc0d8e70da-kube-api-access-xqr79\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:53.989851 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.989762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftvc\" (UniqueName: \"kubernetes.io/projected/6466fa2e-35f0-499b-a564-a878278d3377-kube-api-access-bftvc\") pod \"aws-ebs-csi-driver-node-h6b4k\" (UID: \"6466fa2e-35f0-499b-a564-a878278d3377\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:53.989851 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.989765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghqb\" (UniqueName: \"kubernetes.io/projected/ac8a58c4-152a-421d-8cf6-609cb576d716-kube-api-access-sghqb\") pod \"iptables-alerter-vjhwb\" (UID: \"ac8a58c4-152a-421d-8cf6-609cb576d716\") " pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:53.993186 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.993133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjpw8\" (UniqueName: \"kubernetes.io/projected/9f54592e-2056-4dd4-969c-eaeec3227210-kube-api-access-gjpw8\") pod \"node-ca-ft7jx\" (UID: \"9f54592e-2056-4dd4-969c-eaeec3227210\") " pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:53.993282 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:53.993266 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5ct6\" (UniqueName: \"kubernetes.io/projected/f01f26bd-0336-4941-bc54-62734d65f013-kube-api-access-s5ct6\") pod \"multus-additional-cni-plugins-r2hzb\" (UID: \"f01f26bd-0336-4941-bc54-62734d65f013\") " pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:54.082218 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-systemd\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082218 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082219 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-var-lib-kubelet\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-run-openvswitch\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082265 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-kubernetes\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-sysconfig\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-host\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-slash\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082328 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-systemd\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082347 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-var-lib-kubelet\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-sysctl-conf\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082328 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-run-openvswitch\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082380 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-sysconfig\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082386 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-lib-modules\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082419 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-run-systemd\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.082440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082418 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-slash\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-run-systemd\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082461 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-kubernetes\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-host\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082492 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-sysctl-conf\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082501 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-lib-modules\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-ovnkube-script-lib\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqq4j\" (UniqueName: \"kubernetes.io/projected/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-kube-api-access-jqq4j\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082562 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ffe5e8fe-18fc-43f5-8f36-23b3e89025fa-konnectivity-ca\") pod \"konnectivity-agent-pjtv9\" (UID: \"ffe5e8fe-18fc-43f5-8f36-23b3e89025fa\") " pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082680 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-sysctl-d\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-sys\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082743 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a31ef62-788d-40d4-b735-08587d7e341d-kubelet-config\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a31ef62-788d-40d4-b735-08587d7e341d-dbus\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-systemd-units\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-run-ovn\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5bf475c5-b68d-4235-b53e-ff2420b1e556-hosts-file\") pod \"node-resolver-vq45m\" (UID: \"5bf475c5-b68d-4235-b53e-ff2420b1e556\") " pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:54.083055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082859 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-sys\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpvc\" (UniqueName: \"kubernetes.io/projected/5bf475c5-b68d-4235-b53e-ff2420b1e556-kube-api-access-mfpvc\") pod \"node-resolver-vq45m\" (UID: \"5bf475c5-b68d-4235-b53e-ff2420b1e556\") " pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082943 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-sysctl-d\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082949 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-tuned\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082958 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a31ef62-788d-40d4-b735-08587d7e341d-dbus\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8r5\" (UniqueName: \"kubernetes.io/projected/9dff5577-ae04-4e24-9c72-f2175eb61ff8-kube-api-access-tg8r5\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.082998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-run-netns\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-run\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-systemd-units\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a31ef62-788d-40d4-b735-08587d7e341d-kubelet-config\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-kubelet\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083120 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-kubelet\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.083132 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-run\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.083184 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret podName:8a31ef62-788d-40d4-b735-08587d7e341d nodeName:}" failed. No retries permitted until 2026-09-16 19:04:54.583166498 +0000 UTC m=+3.132223764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret") pod "global-pull-secret-syncer-hjmsb" (UID: "8a31ef62-788d-40d4-b735-08587d7e341d") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083195 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-ovnkube-script-lib\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.083954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083200 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ffe5e8fe-18fc-43f5-8f36-23b3e89025fa-konnectivity-ca\") pod \"konnectivity-agent-pjtv9\" (UID: \"ffe5e8fe-18fc-43f5-8f36-23b3e89025fa\") " pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-etc-openvswitch\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083212 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-run-netns\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083219 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5bf475c5-b68d-4235-b53e-ff2420b1e556-hosts-file\") pod \"node-resolver-vq45m\" (UID: \"5bf475c5-b68d-4235-b53e-ff2420b1e556\") " pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-run-ovn\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083245 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-etc-openvswitch\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-log-socket\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083279 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-log-socket\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-cni-bin\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-cni-netd\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083346 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dff5577-ae04-4e24-9c72-f2175eb61ff8-tmp\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-cni-bin\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-node-log\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083399 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-cni-netd\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-ovn-node-metrics-cert\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083444 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-node-log\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ffe5e8fe-18fc-43f5-8f36-23b3e89025fa-agent-certs\") pod \"konnectivity-agent-pjtv9\" (UID: \"ffe5e8fe-18fc-43f5-8f36-23b3e89025fa\") " pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083481 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-modprobe-d\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.084811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-env-overrides\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083558 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-ovnkube-config\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf475c5-b68d-4235-b53e-ff2420b1e556-tmp-dir\") pod \"node-resolver-vq45m\" (UID: \"5bf475c5-b68d-4235-b53e-ff2420b1e556\") " pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-var-lib-openvswitch\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083668 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-var-lib-openvswitch\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-modprobe-d\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.083780 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.084028 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf475c5-b68d-4235-b53e-ff2420b1e556-tmp-dir\") pod \"node-resolver-vq45m\" (UID: \"5bf475c5-b68d-4235-b53e-ff2420b1e556\") " pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.084244 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-env-overrides\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.084247 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-ovnkube-config\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.085754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.085632 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9dff5577-ae04-4e24-9c72-f2175eb61ff8-etc-tuned\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.086278 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.086223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-ovn-node-metrics-cert\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.086419 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.086401 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ffe5e8fe-18fc-43f5-8f36-23b3e89025fa-agent-certs\") pod \"konnectivity-agent-pjtv9\" (UID: \"ffe5e8fe-18fc-43f5-8f36-23b3e89025fa\") " pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:04:54.086478 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.086461 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dff5577-ae04-4e24-9c72-f2175eb61ff8-tmp\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.092650 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.092626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqq4j\" (UniqueName: \"kubernetes.io/projected/f96bb20f-d067-49b4-b9b3-d5bdbddb07bf-kube-api-access-jqq4j\") pod \"ovnkube-node-w7hqb\" (UID: \"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.092983 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.092963 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpvc\" (UniqueName: \"kubernetes.io/projected/5bf475c5-b68d-4235-b53e-ff2420b1e556-kube-api-access-mfpvc\") pod \"node-resolver-vq45m\" (UID: \"5bf475c5-b68d-4235-b53e-ff2420b1e556\") " pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:54.093137 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.093118 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8r5\" (UniqueName: \"kubernetes.io/projected/9dff5577-ae04-4e24-9c72-f2175eb61ff8-kube-api-access-tg8r5\") pod \"tuned-glj5r\" (UID: \"9dff5577-ae04-4e24-9c72-f2175eb61ff8\") " pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.160994 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.160952 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vjhwb" Sep 16 19:04:54.168853 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.168829 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5tk6q" Sep 16 19:04:54.175891 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:54.175856 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8a58c4_152a_421d_8cf6_609cb576d716.slice/crio-c8341e0246e8fea1a4798f5b837f7512d6350ee7d37f1e0e8138b90e349e36c7 WatchSource:0}: Error finding container c8341e0246e8fea1a4798f5b837f7512d6350ee7d37f1e0e8138b90e349e36c7: Status 404 returned error can't find the container with id c8341e0246e8fea1a4798f5b837f7512d6350ee7d37f1e0e8138b90e349e36c7 Sep 16 19:04:54.181178 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.181155 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ft7jx" Sep 16 19:04:54.181526 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:54.181494 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66049e9_573e_4aa1_add4_b930be25fb66.slice/crio-0957d760c3c0fc6d2019a0c86c14caef95c0b8257f3b272b5869fa0f3268547e WatchSource:0}: Error finding container 0957d760c3c0fc6d2019a0c86c14caef95c0b8257f3b272b5869fa0f3268547e: Status 404 returned error can't find the container with id 0957d760c3c0fc6d2019a0c86c14caef95c0b8257f3b272b5869fa0f3268547e Sep 16 19:04:54.185702 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.185598 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" Sep 16 19:04:54.192902 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.192750 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" Sep 16 19:04:54.193096 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:54.193073 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f54592e_2056_4dd4_969c_eaeec3227210.slice/crio-5d46ad4b571e49845a4cbf92c19d95d5f9a569452eb67c04cec3d454c2c34431 WatchSource:0}: Error finding container 5d46ad4b571e49845a4cbf92c19d95d5f9a569452eb67c04cec3d454c2c34431: Status 404 returned error can't find the container with id 5d46ad4b571e49845a4cbf92c19d95d5f9a569452eb67c04cec3d454c2c34431 Sep 16 19:04:54.197487 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:54.197471 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01f26bd_0336_4941_bc54_62734d65f013.slice/crio-a77e251ae63a9348daea2eb6c8d6495cd4f13e1ec9d6f932ffe98e9b1e56b153 WatchSource:0}: Error finding container a77e251ae63a9348daea2eb6c8d6495cd4f13e1ec9d6f932ffe98e9b1e56b153: Status 404 returned error can't find the container with id a77e251ae63a9348daea2eb6c8d6495cd4f13e1ec9d6f932ffe98e9b1e56b153 Sep 16 19:04:54.201217 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.201199 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:04:54.204434 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:54.204408 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6466fa2e_35f0_499b_a564_a878278d3377.slice/crio-45af1f33048e5a83f16598859a0c5f7f57c9d2afe7d9518e6202ae99347f3cca WatchSource:0}: Error finding container 45af1f33048e5a83f16598859a0c5f7f57c9d2afe7d9518e6202ae99347f3cca: Status 404 returned error can't find the container with id 45af1f33048e5a83f16598859a0c5f7f57c9d2afe7d9518e6202ae99347f3cca Sep 16 19:04:54.208065 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.208043 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-glj5r" Sep 16 19:04:54.212085 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:54.212061 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe5e8fe_18fc_43f5_8f36_23b3e89025fa.slice/crio-a221760400c21378484a4c4019343d7d400fb797fe99f538e4732da1df772905 WatchSource:0}: Error finding container a221760400c21378484a4c4019343d7d400fb797fe99f538e4732da1df772905: Status 404 returned error can't find the container with id a221760400c21378484a4c4019343d7d400fb797fe99f538e4732da1df772905 Sep 16 19:04:54.217359 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:54.217322 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dff5577_ae04_4e24_9c72_f2175eb61ff8.slice/crio-19a169ca9794c2f370db00f2738298e315ad11dfba7f7a7d19d70ed5c132dff8 WatchSource:0}: Error finding container 19a169ca9794c2f370db00f2738298e315ad11dfba7f7a7d19d70ed5c132dff8: Status 404 returned error can't find the container with id 19a169ca9794c2f370db00f2738298e315ad11dfba7f7a7d19d70ed5c132dff8 Sep 16 19:04:54.222746 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.222724 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vq45m" Sep 16 19:04:54.228394 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.228374 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:04:54.232577 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:54.232379 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf475c5_b68d_4235_b53e_ff2420b1e556.slice/crio-1e3bf8d0965e38fd74e6ecbefdf475971bc992d4a1fe30d2bbd4798a226095a9 WatchSource:0}: Error finding container 1e3bf8d0965e38fd74e6ecbefdf475971bc992d4a1fe30d2bbd4798a226095a9: Status 404 returned error can't find the container with id 1e3bf8d0965e38fd74e6ecbefdf475971bc992d4a1fe30d2bbd4798a226095a9 Sep 16 19:04:54.240926 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:04:54.240897 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96bb20f_d067_49b4_b9b3_d5bdbddb07bf.slice/crio-da174f66456257afe6ba982a6ca21059f91cb7aadcf4fc5ddd84b74caa541513 WatchSource:0}: Error finding container da174f66456257afe6ba982a6ca21059f91cb7aadcf4fc5ddd84b74caa541513: Status 404 returned error can't find the container with id da174f66456257afe6ba982a6ca21059f91cb7aadcf4fc5ddd84b74caa541513 Sep 16 19:04:54.486002 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.485920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:04:54.486002 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.485974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:54.486214 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.486107 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:04:54.486214 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.486153 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs podName:10b511a5-508f-4ce7-988a-7abc0d8e70da nodeName:}" failed. No retries permitted until 2026-09-16 19:04:55.486140049 +0000 UTC m=+4.035197300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs") pod "network-metrics-daemon-mzvlv" (UID: "10b511a5-508f-4ce7-988a-7abc0d8e70da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:04:54.486564 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.486441 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:04:54.486564 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.486470 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:04:54.486564 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.486483 2562 projected.go:196] Error preparing data for projected volume kube-api-access-dpjx8 for pod openshift-network-diagnostics/network-check-target-cnrd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:04:54.486564 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.486537 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8 podName:49609a4b-5c7b-412e-b0ae-01d78c0ea546 nodeName:}" failed. No retries permitted until 2026-09-16 19:04:55.486517463 +0000 UTC m=+4.035574718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpjx8" (UniqueName: "kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8") pod "network-check-target-cnrd5" (UID: "49609a4b-5c7b-412e-b0ae-01d78c0ea546") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:04:54.587311 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.587279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:54.587457 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.587437 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:04:54.587523 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:54.587513 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret podName:8a31ef62-788d-40d4-b735-08587d7e341d nodeName:}" failed. No retries permitted until 2026-09-16 19:04:55.587492247 +0000 UTC m=+4.136549519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret") pod "global-pull-secret-syncer-hjmsb" (UID: "8a31ef62-788d-40d4-b735-08587d7e341d") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:04:54.921417 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.921320 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:59:53 +0000 UTC" deadline="2028-05-24 20:01:05.337243288 +0000 UTC" Sep 16 19:04:54.921417 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.921375 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14784h56m10.415874807s" Sep 16 19:04:54.969186 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.969105 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-glj5r" event={"ID":"9dff5577-ae04-4e24-9c72-f2175eb61ff8","Type":"ContainerStarted","Data":"19a169ca9794c2f370db00f2738298e315ad11dfba7f7a7d19d70ed5c132dff8"} Sep 16 19:04:54.971984 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.971950 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pjtv9" event={"ID":"ffe5e8fe-18fc-43f5-8f36-23b3e89025fa","Type":"ContainerStarted","Data":"a221760400c21378484a4c4019343d7d400fb797fe99f538e4732da1df772905"} Sep 16 19:04:54.974178 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.974142 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" event={"ID":"f01f26bd-0336-4941-bc54-62734d65f013","Type":"ContainerStarted","Data":"a77e251ae63a9348daea2eb6c8d6495cd4f13e1ec9d6f932ffe98e9b1e56b153"} Sep 16 19:04:54.976299 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.976272 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ft7jx" event={"ID":"9f54592e-2056-4dd4-969c-eaeec3227210","Type":"ContainerStarted","Data":"5d46ad4b571e49845a4cbf92c19d95d5f9a569452eb67c04cec3d454c2c34431"} Sep 16 19:04:54.981673 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.981306 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5tk6q" event={"ID":"f66049e9-573e-4aa1-add4-b930be25fb66","Type":"ContainerStarted","Data":"0957d760c3c0fc6d2019a0c86c14caef95c0b8257f3b272b5869fa0f3268547e"} Sep 16 19:04:54.983072 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.983042 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vjhwb" event={"ID":"ac8a58c4-152a-421d-8cf6-609cb576d716","Type":"ContainerStarted","Data":"c8341e0246e8fea1a4798f5b837f7512d6350ee7d37f1e0e8138b90e349e36c7"} Sep 16 19:04:54.985186 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.985161 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerStarted","Data":"da174f66456257afe6ba982a6ca21059f91cb7aadcf4fc5ddd84b74caa541513"} Sep 16 19:04:54.986636 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.986612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vq45m" event={"ID":"5bf475c5-b68d-4235-b53e-ff2420b1e556","Type":"ContainerStarted","Data":"1e3bf8d0965e38fd74e6ecbefdf475971bc992d4a1fe30d2bbd4798a226095a9"} Sep 16 19:04:54.994873 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:54.994844 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" event={"ID":"6466fa2e-35f0-499b-a564-a878278d3377","Type":"ContainerStarted","Data":"45af1f33048e5a83f16598859a0c5f7f57c9d2afe7d9518e6202ae99347f3cca"} Sep 16 19:04:55.493156 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:55.492253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:04:55.493156 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:55.492316 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:55.493156 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.492584 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:04:55.493156 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.492607 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:04:55.493156 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.492620 2562 projected.go:196] Error preparing data for projected volume kube-api-access-dpjx8 for pod openshift-network-diagnostics/network-check-target-cnrd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:04:55.493156 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.492676 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8 podName:49609a4b-5c7b-412e-b0ae-01d78c0ea546 nodeName:}" failed. No retries permitted until 2026-09-16 19:04:57.492657663 +0000 UTC m=+6.041714920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpjx8" (UniqueName: "kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8") pod "network-check-target-cnrd5" (UID: "49609a4b-5c7b-412e-b0ae-01d78c0ea546") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:04:55.493156 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.492757 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:04:55.493156 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.492790 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs podName:10b511a5-508f-4ce7-988a-7abc0d8e70da nodeName:}" failed. No retries permitted until 2026-09-16 19:04:57.492778328 +0000 UTC m=+6.041835580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs") pod "network-metrics-daemon-mzvlv" (UID: "10b511a5-508f-4ce7-988a-7abc0d8e70da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:04:55.593940 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:55.593213 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:55.593940 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.593528 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:04:55.593940 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.593591 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret podName:8a31ef62-788d-40d4-b735-08587d7e341d nodeName:}" failed. No retries permitted until 2026-09-16 19:04:57.593573592 +0000 UTC m=+6.142630864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret") pod "global-pull-secret-syncer-hjmsb" (UID: "8a31ef62-788d-40d4-b735-08587d7e341d") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:04:55.921698 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:55.921565 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:59:53 +0000 UTC" deadline="2028-04-30 00:19:41.274257239 +0000 UTC" Sep 16 19:04:55.921698 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:55.921629 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14189h14m45.352633076s" Sep 16 19:04:55.958562 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:55.957630 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:04:55.958562 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.957756 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:04:55.958562 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:55.958197 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:55.958562 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.958362 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:04:55.958562 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:55.958449 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:55.958562 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:55.958522 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:04:57.007601 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:57.007281 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal" event={"ID":"322d0d38fd19fa48489dd88803493fc9","Type":"ContainerStarted","Data":"37140f596deeabb05e6edb74e3ad62904421c0a1940b8d8fa067bdfbca59a964"} Sep 16 19:04:57.525949 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:57.525911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:04:57.526155 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:57.525983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:57.526155 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.526124 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:04:57.526270 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.526184 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs podName:10b511a5-508f-4ce7-988a-7abc0d8e70da nodeName:}" failed. No retries permitted until 2026-09-16 19:05:01.526166418 +0000 UTC m=+10.075223677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs") pod "network-metrics-daemon-mzvlv" (UID: "10b511a5-508f-4ce7-988a-7abc0d8e70da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:04:57.526270 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.526267 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:04:57.526376 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.526285 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:04:57.526376 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.526298 2562 projected.go:196] Error preparing data for projected volume kube-api-access-dpjx8 for pod openshift-network-diagnostics/network-check-target-cnrd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:04:57.526376 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.526332 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8 podName:49609a4b-5c7b-412e-b0ae-01d78c0ea546 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:01.526320757 +0000 UTC m=+10.075378014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpjx8" (UniqueName: "kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8") pod "network-check-target-cnrd5" (UID: "49609a4b-5c7b-412e-b0ae-01d78c0ea546") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:04:57.626628 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:57.626535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:57.626790 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.626641 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:04:57.626790 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.626705 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret podName:8a31ef62-788d-40d4-b735-08587d7e341d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:01.62668622 +0000 UTC m=+10.175743497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret") pod "global-pull-secret-syncer-hjmsb" (UID: "8a31ef62-788d-40d4-b735-08587d7e341d") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:04:57.958504 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:57.958418 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:57.958504 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:57.958483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:04:57.958703 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:57.958543 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:57.958703 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.958664 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:04:57.958816 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.958783 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:04:57.958903 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:57.958883 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:04:58.012109 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:58.012076 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" event={"ID":"06f029b86425ea0c165607a9c36ed11c","Type":"ContainerStarted","Data":"484a7858e6cb9ac93e79c5b98e900d7bc8a314c15377072c58653bba9f39b44d"} Sep 16 19:04:58.028672 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:58.028448 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-17.ec2.internal" podStartSLOduration=5.028429922 podStartE2EDuration="5.028429922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:57.024297504 +0000 UTC m=+5.573354778" watchObservedRunningTime="2026-09-16 19:04:58.028429922 +0000 UTC m=+6.577487200" Sep 16 19:04:59.016221 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:59.016115 2562 generic.go:356] "Generic (PLEG): container finished" podID="06f029b86425ea0c165607a9c36ed11c" containerID="484a7858e6cb9ac93e79c5b98e900d7bc8a314c15377072c58653bba9f39b44d" exitCode=0 Sep 16 19:04:59.016221 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:59.016160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" event={"ID":"06f029b86425ea0c165607a9c36ed11c","Type":"ContainerDied","Data":"484a7858e6cb9ac93e79c5b98e900d7bc8a314c15377072c58653bba9f39b44d"} Sep 16 19:04:59.958789 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:59.957991 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:04:59.958789 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:59.958136 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:04:59.958789 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:59.958709 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:04:59.959116 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:59.958828 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:04:59.959116 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:04:59.958952 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:04:59.959116 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:04:59.959054 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:01.558145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:01.557869 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:01.558632 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.558093 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:01.558632 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.558233 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:01.558632 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.558248 2562 projected.go:196] Error preparing data for projected volume kube-api-access-dpjx8 for pod openshift-network-diagnostics/network-check-target-cnrd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:01.558632 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.558304 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:01.558632 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:01.558201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:01.558632 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.558311 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8 podName:49609a4b-5c7b-412e-b0ae-01d78c0ea546 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:09.558290926 +0000 UTC m=+18.107348189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpjx8" (UniqueName: "kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8") pod "network-check-target-cnrd5" (UID: "49609a4b-5c7b-412e-b0ae-01d78c0ea546") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:01.558632 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.558377 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs podName:10b511a5-508f-4ce7-988a-7abc0d8e70da nodeName:}" failed. No retries permitted until 2026-09-16 19:05:09.558361093 +0000 UTC m=+18.107418352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs") pod "network-metrics-daemon-mzvlv" (UID: "10b511a5-508f-4ce7-988a-7abc0d8e70da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:01.659060 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:01.658904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:01.659211 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.659074 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:01.659211 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.659153 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret podName:8a31ef62-788d-40d4-b735-08587d7e341d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:09.659134159 +0000 UTC m=+18.208191414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret") pod "global-pull-secret-syncer-hjmsb" (UID: "8a31ef62-788d-40d4-b735-08587d7e341d") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:01.958600 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:01.958518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:01.958756 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.958625 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:01.958901 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:01.958874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:01.959044 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:01.958999 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:01.959044 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.959030 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:01.959165 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:01.959104 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:03.961024 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:03.960978 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:03.961490 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:03.961031 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:03.961490 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:03.961034 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:03.961490 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:03.961139 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:03.961490 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:03.961261 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:03.961490 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:03.961349 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:05.960523 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:05.960489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:05.960968 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:05.960598 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:05.960968 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:05.960607 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:05.960968 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:05.960698 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:05.960968 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:05.960696 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:05.960968 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:05.960776 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:07.957416 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:07.957377 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:07.957864 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:07.957455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:07.957864 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:07.957474 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:07.957864 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:07.957591 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:07.957864 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:07.957644 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:07.957864 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:07.957719 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:09.619228 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:09.619189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:09.619749 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:09.619251 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:09.619749 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.619357 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:09.619749 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.619360 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:09.619749 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.619384 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:09.619749 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.619396 2562 projected.go:196] Error preparing data for projected volume kube-api-access-dpjx8 for pod openshift-network-diagnostics/network-check-target-cnrd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:09.619749 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.619423 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs podName:10b511a5-508f-4ce7-988a-7abc0d8e70da nodeName:}" failed. No retries permitted until 2026-09-16 19:05:25.619405746 +0000 UTC m=+34.168463020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs") pod "network-metrics-daemon-mzvlv" (UID: "10b511a5-508f-4ce7-988a-7abc0d8e70da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:09.619749 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.619447 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8 podName:49609a4b-5c7b-412e-b0ae-01d78c0ea546 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:25.619439479 +0000 UTC m=+34.168496736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpjx8" (UniqueName: "kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8") pod "network-check-target-cnrd5" (UID: "49609a4b-5c7b-412e-b0ae-01d78c0ea546") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:09.719916 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:09.719877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:09.720110 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.720053 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:09.720161 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.720130 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret podName:8a31ef62-788d-40d4-b735-08587d7e341d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:25.720113787 +0000 UTC m=+34.269171041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret") pod "global-pull-secret-syncer-hjmsb" (UID: "8a31ef62-788d-40d4-b735-08587d7e341d") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:09.960311 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:09.960234 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:09.960473 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:09.960237 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:09.960473 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.960350 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:09.960473 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:09.960366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:09.960473 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.960457 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:09.960658 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:09.960546 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:11.963094 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:11.963060 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:11.963094 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:11.963093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:11.963579 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:11.963091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:11.963579 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:11.963170 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:11.963579 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:11.963301 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:11.963579 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:11.963405 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:13.048385 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.048146 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" event={"ID":"06f029b86425ea0c165607a9c36ed11c","Type":"ContainerStarted","Data":"b61ccc3d10c63c96ac397d3c0d16bcc7b9a73978a96da9fea44e35e7a6332cab"} Sep 16 19:05:13.049753 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.049732 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-glj5r" event={"ID":"9dff5577-ae04-4e24-9c72-f2175eb61ff8","Type":"ContainerStarted","Data":"9fd7b5c657591eb038c41fc7475d9e585972dce6200a14cefd6b24839a1cbba0"} Sep 16 19:05:13.051168 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.051143 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pjtv9" event={"ID":"ffe5e8fe-18fc-43f5-8f36-23b3e89025fa","Type":"ContainerStarted","Data":"f5133ad4f5ad1657823635a0df231e28650d7113638a4eb63ca90d78b1f0cf73"} Sep 16 19:05:13.052591 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.052569 2562 generic.go:356] "Generic (PLEG): container finished" podID="f01f26bd-0336-4941-bc54-62734d65f013" containerID="509e76e56680a42f846e6c79d0725ad1e17eed1ac50a35935688bd31a5c2b883" exitCode=0 Sep 16 19:05:13.052671 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.052647 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" event={"ID":"f01f26bd-0336-4941-bc54-62734d65f013","Type":"ContainerDied","Data":"509e76e56680a42f846e6c79d0725ad1e17eed1ac50a35935688bd31a5c2b883"} Sep 16 19:05:13.054344 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.054083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ft7jx" event={"ID":"9f54592e-2056-4dd4-969c-eaeec3227210","Type":"ContainerStarted","Data":"457e4f4ff3113d1e58931165836455bf29a0a458452657fb4dde61bc985710ad"} Sep 16 19:05:13.055536 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.055516 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5tk6q" event={"ID":"f66049e9-573e-4aa1-add4-b930be25fb66","Type":"ContainerStarted","Data":"fdf025f542039be5956d803ee872233c186c6013b124e0990c6a2a4166b33794"} Sep 16 19:05:13.058248 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.058233 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:05:13.058608 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.058587 2562 generic.go:356] "Generic (PLEG): container finished" podID="f96bb20f-d067-49b4-b9b3-d5bdbddb07bf" containerID="678806c751c21baa7eef7c542da02078faf9acf8c26449d9fba3a129d4845b75" exitCode=1 Sep 16 19:05:13.058718 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.058645 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerStarted","Data":"08c06316fb3c2a3539bf047cb72c4e9e21a93b4a7a5a7307ebb556c8163ad5bd"} Sep 16 19:05:13.058718 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.058693 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerStarted","Data":"0b6c1e01e9ee43376b9f9ebe1c1f3c6bee9657a0ed111da6ae9ce9ba49cea8e5"} Sep 16 19:05:13.058798 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.058724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerStarted","Data":"71b0a4cf28d2aca912a0bd790821cf28ec9da4a42e2481dc46fa16c9ae577ccb"} Sep 16 19:05:13.058798 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.058751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerStarted","Data":"643e36d5e6814284253272eba72e085d76645711d9c27c37e0abb0f3487d7a37"} Sep 16 19:05:13.058798 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.058779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerDied","Data":"678806c751c21baa7eef7c542da02078faf9acf8c26449d9fba3a129d4845b75"} Sep 16 19:05:13.058886 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.058808 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerStarted","Data":"3bb7bd4a826d94e007f2497c2ef026ce46ab25c1913ec615bb538615cffb205c"} Sep 16 19:05:13.059966 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.059948 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vq45m" event={"ID":"5bf475c5-b68d-4235-b53e-ff2420b1e556","Type":"ContainerStarted","Data":"bbdbac6a1d2056a0dda39a17bcfa051fc8c733e75ce4257711979ba5a39fd5d6"} Sep 16 19:05:13.061344 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.061327 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" event={"ID":"6466fa2e-35f0-499b-a564-a878278d3377","Type":"ContainerStarted","Data":"cb058a8b85572c17bedc589dc439c7afa26c0d5da5284318f0230549d1285220"} Sep 16 19:05:13.091850 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.091809 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-17.ec2.internal" podStartSLOduration=20.091796709 podStartE2EDuration="20.091796709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:13.09139755 +0000 UTC m=+21.640454823" watchObservedRunningTime="2026-09-16 19:05:13.091796709 +0000 UTC m=+21.640853983" Sep 16 19:05:13.145002 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.144956 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-glj5r" podStartSLOduration=3.231327553 podStartE2EDuration="21.144941074s" totalImagesPullingTime="17.913613521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:13.117996747 +0000 UTC m=+21.667054022" watchObservedRunningTime="2026-09-16 19:05:13.144941074 +0000 UTC m=+21.693998349" Sep 16 19:05:13.145262 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.145086 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vq45m" podStartSLOduration=3.248516351 podStartE2EDuration="21.145079198s" totalImagesPullingTime="17.896562847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:13.145034518 +0000 UTC m=+21.694091783" watchObservedRunningTime="2026-09-16 19:05:13.145079198 +0000 UTC m=+21.694136474" Sep 16 19:05:13.174069 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.174005 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5tk6q" podStartSLOduration=2.665245623 podStartE2EDuration="21.173989633s" totalImagesPullingTime="18.50874401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:13.173917592 +0000 UTC m=+21.722974866" watchObservedRunningTime="2026-09-16 19:05:13.173989633 +0000 UTC m=+21.723046906" Sep 16 19:05:13.248610 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.248559 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ft7jx" podStartSLOduration=3.310909438 podStartE2EDuration="21.248543699s" totalImagesPullingTime="17.937634261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:13.20714611 +0000 UTC m=+21.756203384" watchObservedRunningTime="2026-09-16 19:05:13.248543699 +0000 UTC m=+21.797600970" Sep 16 19:05:13.634323 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.634297 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 19:05:13.934175 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.934063 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T19:05:13.634319248Z","UUID":"9b49e0f0-5600-4fc1-a437-9e43bc51752d","Handler":null,"Name":"","Endpoint":""} Sep 16 19:05:13.935538 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.935519 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 19:05:13.935651 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.935552 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 19:05:13.957335 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.957302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:13.957335 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.957331 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:13.957525 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:13.957302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:13.957525 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:13.957434 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:13.957593 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:13.957548 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:13.957631 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:13.957618 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:14.065332 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:14.065301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vjhwb" event={"ID":"ac8a58c4-152a-421d-8cf6-609cb576d716","Type":"ContainerStarted","Data":"13bdb97f42d85e8a3faca61b1d17d874e8e7ea5a7f419794775e577191a00f2c"} Sep 16 19:05:14.067165 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:14.067135 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" event={"ID":"6466fa2e-35f0-499b-a564-a878278d3377","Type":"ContainerStarted","Data":"54d65a12bdbeb997bf1074f6271d6a0d090c8e1195794915cd0aaf586df6ade0"} Sep 16 19:05:14.087700 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:14.087638 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pjtv9" podStartSLOduration=4.168783678 podStartE2EDuration="22.087617006s" totalImagesPullingTime="17.918833328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:13.277220237 +0000 UTC m=+21.826277512" watchObservedRunningTime="2026-09-16 19:05:14.087617006 +0000 UTC m=+22.636674281" Sep 16 19:05:14.089843 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:14.088312 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vjhwb" podStartSLOduration=4.133811006 podStartE2EDuration="22.088299069s" totalImagesPullingTime="17.954488063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:14.087304537 +0000 UTC m=+22.636361812" watchObservedRunningTime="2026-09-16 19:05:14.088299069 +0000 UTC m=+22.637356344" Sep 16 19:05:15.074348 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:15.074281 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:05:15.074737 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:15.074712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerStarted","Data":"7ff640345a4468b5190003ddf76d7be24ddbcb1af6e4293b3d9241d5d3666cbc"} Sep 16 19:05:15.076785 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:15.076757 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" event={"ID":"6466fa2e-35f0-499b-a564-a878278d3377","Type":"ContainerStarted","Data":"3c69b0f4b3850630b9817ddacaef89c2d87fa5b85845fc85fc01dc220f8e89ab"} Sep 16 19:05:15.099587 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:15.099524 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6b4k" podStartSLOduration=2.906050514 podStartE2EDuration="23.099503898s" totalImagesPullingTime="20.193453384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:15.097675128 +0000 UTC m=+23.646732403" watchObservedRunningTime="2026-09-16 19:05:15.099503898 +0000 UTC m=+23.648561176" Sep 16 19:05:15.957569 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:15.957538 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:15.957855 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:15.957538 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:15.957855 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:15.957668 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:15.957855 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:15.957759 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:15.957855 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:15.957538 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:15.957855 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:15.957846 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:17.885256 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:17.885067 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:05:17.885778 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:17.885751 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:05:17.958027 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:17.957995 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:17.958208 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:17.958037 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:17.958208 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:17.958136 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:17.958326 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:17.958229 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:17.958326 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:17.958284 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:17.958419 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:17.958357 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:18.084092 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:18.084055 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:05:18.084580 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:18.084550 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pjtv9" Sep 16 19:05:19.087859 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:19.087666 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" event={"ID":"f01f26bd-0336-4941-bc54-62734d65f013","Type":"ContainerStarted","Data":"e7d1a6a6e1246ae6fb1d00a2195dd9059a20e3234ae314583399539f153fd74a"} Sep 16 19:05:19.091295 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:19.091275 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:05:19.091753 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:19.091725 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerStarted","Data":"7e2a8f49c3932d5cd0bf44b23063b1e245bf876855d96b0d67dbf396733bea88"} Sep 16 19:05:19.092055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:19.092036 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:05:19.092191 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:19.092161 2562 scope.go:117] "RemoveContainer" containerID="678806c751c21baa7eef7c542da02078faf9acf8c26449d9fba3a129d4845b75" Sep 16 19:05:19.109567 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:19.109505 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:05:19.957771 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:19.957725 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:19.957771 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:19.957774 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:19.958133 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:19.957743 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:19.958133 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:19.957853 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:19.958133 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:19.957906 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:19.958133 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:19.957961 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:20.101123 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.101101 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:05:20.101632 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.101455 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" event={"ID":"f96bb20f-d067-49b4-b9b3-d5bdbddb07bf","Type":"ContainerStarted","Data":"3e48a83e0ffc084877e00e0f8fa2184cd0246c78b64720247dd56cbb409873db"} Sep 16 19:05:20.101632 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.101523 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 16 19:05:20.101828 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.101811 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:05:20.103455 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.103433 2562 generic.go:356] "Generic (PLEG): container finished" podID="f01f26bd-0336-4941-bc54-62734d65f013" containerID="e7d1a6a6e1246ae6fb1d00a2195dd9059a20e3234ae314583399539f153fd74a" exitCode=0 Sep 16 19:05:20.103565 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.103515 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" event={"ID":"f01f26bd-0336-4941-bc54-62734d65f013","Type":"ContainerDied","Data":"e7d1a6a6e1246ae6fb1d00a2195dd9059a20e3234ae314583399539f153fd74a"} Sep 16 19:05:20.116662 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.116637 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:05:20.226879 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.226785 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" podStartSLOduration=10.296940752 podStartE2EDuration="28.226770109s" totalImagesPullingTime="17.929829357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:20.162062354 +0000 UTC m=+28.711119629" watchObservedRunningTime="2026-09-16 19:05:20.226770109 +0000 UTC m=+28.775827382" Sep 16 19:05:20.892970 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.892927 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mzvlv"] Sep 16 19:05:20.893157 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.893091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:20.893241 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:20.893219 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:20.895888 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.895850 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cnrd5"] Sep 16 19:05:20.896043 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.895955 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:20.896118 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:20.896080 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:20.908668 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.908637 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hjmsb"] Sep 16 19:05:20.908800 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:20.908741 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:20.908897 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:20.908876 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:21.105992 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:21.105960 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 16 19:05:21.207353 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:21.207184 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:05:22.110630 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:22.110594 2562 generic.go:356] "Generic (PLEG): container finished" podID="f01f26bd-0336-4941-bc54-62734d65f013" containerID="c419f6331bdbbd8168faab6fcbbeab00545a96a02b286946953cc23bef6d0c97" exitCode=0 Sep 16 19:05:22.111098 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:22.110667 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" event={"ID":"f01f26bd-0336-4941-bc54-62734d65f013","Type":"ContainerDied","Data":"c419f6331bdbbd8168faab6fcbbeab00545a96a02b286946953cc23bef6d0c97"} Sep 16 19:05:22.957208 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:22.957179 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:22.957208 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:22.957207 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:22.957397 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:22.957207 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:22.957397 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:22.957308 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:22.957397 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:22.957382 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:22.957547 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:22.957462 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:23.115648 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:23.115609 2562 generic.go:356] "Generic (PLEG): container finished" podID="f01f26bd-0336-4941-bc54-62734d65f013" containerID="62e23da6d9df101eded471e4d6260411ab23d0e3cf4632e8526a548d56a9fc8b" exitCode=0 Sep 16 19:05:23.116033 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:23.115670 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" event={"ID":"f01f26bd-0336-4941-bc54-62734d65f013","Type":"ContainerDied","Data":"62e23da6d9df101eded471e4d6260411ab23d0e3cf4632e8526a548d56a9fc8b"} Sep 16 19:05:23.126186 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:23.126154 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" podUID="f96bb20f-d067-49b4-b9b3-d5bdbddb07bf" containerName="ovnkube-controller" probeResult="failure" output="" Sep 16 19:05:24.957822 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:24.957790 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:24.958201 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:24.957790 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:24.958201 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:24.957890 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjmsb" podUID="8a31ef62-788d-40d4-b735-08587d7e341d" Sep 16 19:05:24.958201 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:24.957790 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:24.958201 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:24.957956 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cnrd5" podUID="49609a4b-5c7b-412e-b0ae-01d78c0ea546" Sep 16 19:05:24.958201 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:24.958050 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:05:25.266348 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.266272 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-17.ec2.internal" event="NodeReady" Sep 16 19:05:25.266501 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.266432 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 19:05:25.312900 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.312872 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk"] Sep 16 19:05:25.316499 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.316485 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.319472 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.319285 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 19:05:25.319472 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.319301 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 16 19:05:25.319769 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.319755 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 19:05:25.320096 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.320082 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 19:05:25.321388 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.321183 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb"] Sep 16 19:05:25.325365 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.325346 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75bf757ff8-cknd2"] Sep 16 19:05:25.325471 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.325450 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.329196 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.329180 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62"] Sep 16 19:05:25.329319 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.329300 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.330071 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.330054 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 16 19:05:25.330338 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.330320 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 16 19:05:25.330939 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.330924 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 16 19:05:25.331734 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.331716 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 16 19:05:25.332825 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.332810 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" Sep 16 19:05:25.334918 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.334899 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 19:05:25.335333 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.335316 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 19:05:25.335381 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.335343 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 19:05:25.335968 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.335944 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk"] Sep 16 19:05:25.336427 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.336410 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 16 19:05:25.336747 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.336733 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mchdw\"" Sep 16 19:05:25.337457 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.337443 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pps2s\"" Sep 16 19:05:25.339909 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.339889 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fjz6k"] Sep 16 19:05:25.343700 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.343682 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.346719 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.346700 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 19:05:25.346809 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.346723 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 19:05:25.347983 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.347963 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pvzq\"" Sep 16 19:05:25.349742 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.349415 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb"] Sep 16 19:05:25.357241 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.357219 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 19:05:25.357371 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.357348 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62"] Sep 16 19:05:25.357836 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.357821 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w5tdv"] Sep 16 19:05:25.361065 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.361049 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w5tdv" Sep 16 19:05:25.365574 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.365555 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 19:05:25.365912 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.365884 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d25ds\"" Sep 16 19:05:25.367506 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.367209 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 19:05:25.367851 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.367707 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 19:05:25.367953 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.367932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fjz6k"] Sep 16 19:05:25.368822 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.368801 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75bf757ff8-cknd2"] Sep 16 19:05:25.381262 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.381242 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w5tdv"] Sep 16 19:05:25.429177 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be43e7da-de98-4725-9999-00f73403f065-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67774866f8-fhc62\" (UID: \"be43e7da-de98-4725-9999-00f73403f065\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" Sep 16 19:05:25.429316 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429185 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhh5\" (UniqueName: \"kubernetes.io/projected/38a44740-3ffc-48fc-9895-637cf651e633-kube-api-access-rwhh5\") pod \"klusterlet-addon-workmgr-5bfbd9c499-fnmsk\" (UID: \"38a44740-3ffc-48fc-9895-637cf651e633\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.429316 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429212 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vzg\" (UniqueName: \"kubernetes.io/projected/be43e7da-de98-4725-9999-00f73403f065-kube-api-access-w7vzg\") pod \"managed-serviceaccount-addon-agent-67774866f8-fhc62\" (UID: \"be43e7da-de98-4725-9999-00f73403f065\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" Sep 16 19:05:25.429316 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429269 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-hub\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.429316 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38a44740-3ffc-48fc-9895-637cf651e633-tmp\") pod \"klusterlet-addon-workmgr-5bfbd9c499-fnmsk\" (UID: \"38a44740-3ffc-48fc-9895-637cf651e633\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.429512 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-ca\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.429512 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/38a44740-3ffc-48fc-9895-637cf651e633-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bfbd9c499-fnmsk\" (UID: \"38a44740-3ffc-48fc-9895-637cf651e633\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.429512 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-bound-sa-token\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.429512 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429442 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.429512 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bsnl\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-kube-api-access-4bsnl\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.429512 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429504 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjst\" (UniqueName: \"kubernetes.io/projected/727aee0d-4e52-4e82-a07d-f7eed7bcc118-kube-api-access-qsjst\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.429740 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429550 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xfv\" (UniqueName: \"kubernetes.io/projected/fb283ce1-2ad2-4616-b24f-9560df64c816-kube-api-access-w2xfv\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.429740 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-registry-certificates\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.429740 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-image-registry-private-configuration\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.429740 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429620 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.429740 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429660 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb283ce1-2ad2-4616-b24f-9560df64c816-tmp-dir\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.429740 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb283ce1-2ad2-4616-b24f-9560df64c816-config-volume\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.429740 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/727aee0d-4e52-4e82-a07d-f7eed7bcc118-additional-service-ca\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.430078 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4162c94c-63b2-42a9-9906-869b44645018-ca-trust-extracted\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.430078 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429802 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-installation-pull-secrets\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.430078 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429838 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.430078 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.430078 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.429939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-trusted-ca\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.530911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.530833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bsnl\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-kube-api-access-4bsnl\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.530911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.530870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjst\" (UniqueName: \"kubernetes.io/projected/727aee0d-4e52-4e82-a07d-f7eed7bcc118-kube-api-access-qsjst\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.530911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.530905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xfv\" (UniqueName: \"kubernetes.io/projected/fb283ce1-2ad2-4616-b24f-9560df64c816-kube-api-access-w2xfv\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.531187 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.530933 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-registry-certificates\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.531187 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-image-registry-private-configuration\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.531187 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.531187 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531131 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb283ce1-2ad2-4616-b24f-9560df64c816-tmp-dir\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.531187 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531169 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb283ce1-2ad2-4616-b24f-9560df64c816-config-volume\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/727aee0d-4e52-4e82-a07d-f7eed7bcc118-additional-service-ca\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4162c94c-63b2-42a9-9906-869b44645018-ca-trust-extracted\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531251 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-installation-pull-secrets\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.531264 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.531282 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf757ff8-cknd2: secret "image-registry-tls" not found Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.531336 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls podName:4162c94c-63b2-42a9-9906-869b44645018 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:26.03131719 +0000 UTC m=+34.580374453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls") pod "image-registry-75bf757ff8-cknd2" (UID: "4162c94c-63b2-42a9-9906-869b44645018") : secret "image-registry-tls" not found Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qt7\" (UniqueName: \"kubernetes.io/projected/2a0bfe9b-6279-4c42-9568-812c00a190e8-kube-api-access-85qt7\") pod \"ingress-canary-w5tdv\" (UID: \"2a0bfe9b-6279-4c42-9568-812c00a190e8\") " pod="openshift-ingress-canary/ingress-canary-w5tdv" Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531450 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.531489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-trusted-ca\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.531957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be43e7da-de98-4725-9999-00f73403f065-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67774866f8-fhc62\" (UID: \"be43e7da-de98-4725-9999-00f73403f065\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" Sep 16 19:05:25.531957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531507 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb283ce1-2ad2-4616-b24f-9560df64c816-tmp-dir\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.531957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a0bfe9b-6279-4c42-9568-812c00a190e8-cert\") pod \"ingress-canary-w5tdv\" (UID: \"2a0bfe9b-6279-4c42-9568-812c00a190e8\") " pod="openshift-ingress-canary/ingress-canary-w5tdv" Sep 16 19:05:25.531957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhh5\" (UniqueName: \"kubernetes.io/projected/38a44740-3ffc-48fc-9895-637cf651e633-kube-api-access-rwhh5\") pod \"klusterlet-addon-workmgr-5bfbd9c499-fnmsk\" (UID: \"38a44740-3ffc-48fc-9895-637cf651e633\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.531957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531584 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-registry-certificates\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.531957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vzg\" (UniqueName: \"kubernetes.io/projected/be43e7da-de98-4725-9999-00f73403f065-kube-api-access-w7vzg\") pod \"managed-serviceaccount-addon-agent-67774866f8-fhc62\" (UID: \"be43e7da-de98-4725-9999-00f73403f065\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" Sep 16 19:05:25.531957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531900 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-hub\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.531957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38a44740-3ffc-48fc-9895-637cf651e633-tmp\") pod \"klusterlet-addon-workmgr-5bfbd9c499-fnmsk\" (UID: \"38a44740-3ffc-48fc-9895-637cf651e633\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.532378 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.531976 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-ca\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.532378 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.532002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/38a44740-3ffc-48fc-9895-637cf651e633-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bfbd9c499-fnmsk\" (UID: \"38a44740-3ffc-48fc-9895-637cf651e633\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.532378 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.532061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-bound-sa-token\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.532378 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.532090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.532378 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.532209 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/727aee0d-4e52-4e82-a07d-f7eed7bcc118-additional-service-ca\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.532378 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.532301 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb283ce1-2ad2-4616-b24f-9560df64c816-config-volume\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.532723 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.532605 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4162c94c-63b2-42a9-9906-869b44645018-ca-trust-extracted\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.532723 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.532621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-trusted-ca\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.532723 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.532713 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:25.532876 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.532773 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls podName:fb283ce1-2ad2-4616-b24f-9560df64c816 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:26.032754321 +0000 UTC m=+34.581811587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls") pod "dns-default-fjz6k" (UID: "fb283ce1-2ad2-4616-b24f-9560df64c816") : secret "dns-default-metrics-tls" not found Sep 16 19:05:25.533171 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.533028 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38a44740-3ffc-48fc-9895-637cf651e633-tmp\") pod \"klusterlet-addon-workmgr-5bfbd9c499-fnmsk\" (UID: \"38a44740-3ffc-48fc-9895-637cf651e633\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.536627 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.536578 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-ca\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.536627 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.536605 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-installation-pull-secrets\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.536871 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.536634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-hub\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.536871 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.536688 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-image-registry-private-configuration\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.536871 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.536698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/38a44740-3ffc-48fc-9895-637cf651e633-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bfbd9c499-fnmsk\" (UID: \"38a44740-3ffc-48fc-9895-637cf651e633\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.536871 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.536821 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.537060 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.536877 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be43e7da-de98-4725-9999-00f73403f065-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67774866f8-fhc62\" (UID: \"be43e7da-de98-4725-9999-00f73403f065\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" Sep 16 19:05:25.537376 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.537352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/727aee0d-4e52-4e82-a07d-f7eed7bcc118-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.538314 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.538289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:25.548622 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.548600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjst\" (UniqueName: \"kubernetes.io/projected/727aee0d-4e52-4e82-a07d-f7eed7bcc118-kube-api-access-qsjst\") pod \"cluster-proxy-proxy-agent-586554554b-4qmzb\" (UID: \"727aee0d-4e52-4e82-a07d-f7eed7bcc118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.548883 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.548857 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhh5\" (UniqueName: \"kubernetes.io/projected/38a44740-3ffc-48fc-9895-637cf651e633-kube-api-access-rwhh5\") pod \"klusterlet-addon-workmgr-5bfbd9c499-fnmsk\" (UID: \"38a44740-3ffc-48fc-9895-637cf651e633\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.549188 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.549168 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xfv\" (UniqueName: \"kubernetes.io/projected/fb283ce1-2ad2-4616-b24f-9560df64c816-kube-api-access-w2xfv\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:25.550999 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.550977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bsnl\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-kube-api-access-4bsnl\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.553155 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.553134 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vzg\" (UniqueName: \"kubernetes.io/projected/be43e7da-de98-4725-9999-00f73403f065-kube-api-access-w7vzg\") pod \"managed-serviceaccount-addon-agent-67774866f8-fhc62\" (UID: \"be43e7da-de98-4725-9999-00f73403f065\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" Sep 16 19:05:25.556261 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.556242 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-bound-sa-token\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:25.627451 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.627416 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:25.632716 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.632691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:25.632844 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.632752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:25.632844 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.632778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qt7\" (UniqueName: \"kubernetes.io/projected/2a0bfe9b-6279-4c42-9568-812c00a190e8-kube-api-access-85qt7\") pod \"ingress-canary-w5tdv\" (UID: \"2a0bfe9b-6279-4c42-9568-812c00a190e8\") " pod="openshift-ingress-canary/ingress-canary-w5tdv" Sep 16 19:05:25.632844 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.632837 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a0bfe9b-6279-4c42-9568-812c00a190e8-cert\") pod \"ingress-canary-w5tdv\" (UID: \"2a0bfe9b-6279-4c42-9568-812c00a190e8\") " pod="openshift-ingress-canary/ingress-canary-w5tdv" Sep 16 19:05:25.632994 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.632892 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:25.632994 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.632920 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:25.632994 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.632933 2562 projected.go:196] Error preparing data for projected volume kube-api-access-dpjx8 for pod openshift-network-diagnostics/network-check-target-cnrd5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:25.632994 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.632956 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:25.632994 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.632993 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8 podName:49609a4b-5c7b-412e-b0ae-01d78c0ea546 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:57.632976087 +0000 UTC m=+66.182033339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpjx8" (UniqueName: "kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8") pod "network-check-target-cnrd5" (UID: "49609a4b-5c7b-412e-b0ae-01d78c0ea546") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:25.633289 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.633059 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs podName:10b511a5-508f-4ce7-988a-7abc0d8e70da nodeName:}" failed. No retries permitted until 2026-09-16 19:05:57.633022476 +0000 UTC m=+66.182079939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs") pod "network-metrics-daemon-mzvlv" (UID: "10b511a5-508f-4ce7-988a-7abc0d8e70da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:25.635614 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.635591 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a0bfe9b-6279-4c42-9568-812c00a190e8-cert\") pod \"ingress-canary-w5tdv\" (UID: \"2a0bfe9b-6279-4c42-9568-812c00a190e8\") " pod="openshift-ingress-canary/ingress-canary-w5tdv" Sep 16 19:05:25.640323 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.640293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:05:25.642834 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.642811 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qt7\" (UniqueName: \"kubernetes.io/projected/2a0bfe9b-6279-4c42-9568-812c00a190e8-kube-api-access-85qt7\") pod \"ingress-canary-w5tdv\" (UID: \"2a0bfe9b-6279-4c42-9568-812c00a190e8\") " pod="openshift-ingress-canary/ingress-canary-w5tdv" Sep 16 19:05:25.655590 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.655557 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" Sep 16 19:05:25.676586 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.676516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:25.682647 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.682482 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w5tdv" Sep 16 19:05:25.734081 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.733494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:25.734081 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.733727 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:25.734081 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:25.733802 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret podName:8a31ef62-788d-40d4-b735-08587d7e341d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:57.733782432 +0000 UTC m=+66.282839704 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret") pod "global-pull-secret-syncer-hjmsb" (UID: "8a31ef62-788d-40d4-b735-08587d7e341d") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:25.919763 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.919732 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk"] Sep 16 19:05:25.923770 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:05:25.923744 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a44740_3ffc_48fc_9895_637cf651e633.slice/crio-d83fae090c3d4cd865b7f8274ae64e84d8775810d1ed9637aff863b8a3ce7962 WatchSource:0}: Error finding container d83fae090c3d4cd865b7f8274ae64e84d8775810d1ed9637aff863b8a3ce7962: Status 404 returned error can't find the container with id d83fae090c3d4cd865b7f8274ae64e84d8775810d1ed9637aff863b8a3ce7962 Sep 16 19:05:25.931436 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.931410 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb"] Sep 16 19:05:25.935117 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:05:25.935089 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod727aee0d_4e52_4e82_a07d_f7eed7bcc118.slice/crio-74dcb41e16570781eed70368c37eb0593c428e589346efdbeb9080ebd62c732f WatchSource:0}: Error finding container 74dcb41e16570781eed70368c37eb0593c428e589346efdbeb9080ebd62c732f: Status 404 returned error can't find the container with id 74dcb41e16570781eed70368c37eb0593c428e589346efdbeb9080ebd62c732f Sep 16 19:05:25.961486 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.961464 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62"] Sep 16 19:05:25.962824 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:05:25.962800 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe43e7da_de98_4725_9999_00f73403f065.slice/crio-f4869e8a7eb816af48fcf01c0448d2e7aeb13dc531345b528dedf36e264063be WatchSource:0}: Error finding container f4869e8a7eb816af48fcf01c0448d2e7aeb13dc531345b528dedf36e264063be: Status 404 returned error can't find the container with id f4869e8a7eb816af48fcf01c0448d2e7aeb13dc531345b528dedf36e264063be Sep 16 19:05:25.983829 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:25.983801 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w5tdv"] Sep 16 19:05:25.988334 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:05:25.988304 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0bfe9b_6279_4c42_9568_812c00a190e8.slice/crio-22001c6f796c54f58b3bbfc23545d7d859c5e9fd2f54ed5569416f50d2788554 WatchSource:0}: Error finding container 22001c6f796c54f58b3bbfc23545d7d859c5e9fd2f54ed5569416f50d2788554: Status 404 returned error can't find the container with id 22001c6f796c54f58b3bbfc23545d7d859c5e9fd2f54ed5569416f50d2788554 Sep 16 19:05:26.036625 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.036590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:26.036756 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:26.036734 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:05:26.036756 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:26.036752 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf757ff8-cknd2: secret "image-registry-tls" not found Sep 16 19:05:26.036841 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.036762 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:26.036841 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:26.036806 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls podName:4162c94c-63b2-42a9-9906-869b44645018 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:27.036788807 +0000 UTC m=+35.585846076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls") pod "image-registry-75bf757ff8-cknd2" (UID: "4162c94c-63b2-42a9-9906-869b44645018") : secret "image-registry-tls" not found Sep 16 19:05:26.036945 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:26.036860 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:26.036945 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:26.036894 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls podName:fb283ce1-2ad2-4616-b24f-9560df64c816 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:27.036885691 +0000 UTC m=+35.585942953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls") pod "dns-default-fjz6k" (UID: "fb283ce1-2ad2-4616-b24f-9560df64c816") : secret "dns-default-metrics-tls" not found Sep 16 19:05:26.125470 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.125439 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" event={"ID":"727aee0d-4e52-4e82-a07d-f7eed7bcc118","Type":"ContainerStarted","Data":"74dcb41e16570781eed70368c37eb0593c428e589346efdbeb9080ebd62c732f"} Sep 16 19:05:26.130181 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.130149 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w5tdv" event={"ID":"2a0bfe9b-6279-4c42-9568-812c00a190e8","Type":"ContainerStarted","Data":"22001c6f796c54f58b3bbfc23545d7d859c5e9fd2f54ed5569416f50d2788554"} Sep 16 19:05:26.131491 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.131463 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" event={"ID":"be43e7da-de98-4725-9999-00f73403f065","Type":"ContainerStarted","Data":"f4869e8a7eb816af48fcf01c0448d2e7aeb13dc531345b528dedf36e264063be"} Sep 16 19:05:26.132674 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.132652 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" event={"ID":"38a44740-3ffc-48fc-9895-637cf651e633","Type":"ContainerStarted","Data":"d83fae090c3d4cd865b7f8274ae64e84d8775810d1ed9637aff863b8a3ce7962"} Sep 16 19:05:26.957525 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.957491 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:26.957774 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.957756 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:26.958066 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.957756 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:26.962300 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.962104 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjb6w\"" Sep 16 19:05:26.964833 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.963214 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 19:05:26.964833 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.963421 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 19:05:26.964833 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.963637 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 19:05:26.964833 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.963805 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 19:05:26.964833 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:26.964052 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45r44\"" Sep 16 19:05:27.046555 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:27.046520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:27.046734 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:27.046614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:27.046832 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:27.046807 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:27.046945 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:27.046929 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls podName:fb283ce1-2ad2-4616-b24f-9560df64c816 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:29.046858888 +0000 UTC m=+37.595916142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls") pod "dns-default-fjz6k" (UID: "fb283ce1-2ad2-4616-b24f-9560df64c816") : secret "dns-default-metrics-tls" not found Sep 16 19:05:27.047414 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:27.047385 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:05:27.047519 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:27.047415 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf757ff8-cknd2: secret "image-registry-tls" not found Sep 16 19:05:27.047519 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:27.047473 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls podName:4162c94c-63b2-42a9-9906-869b44645018 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:29.047456758 +0000 UTC m=+37.596514013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls") pod "image-registry-75bf757ff8-cknd2" (UID: "4162c94c-63b2-42a9-9906-869b44645018") : secret "image-registry-tls" not found Sep 16 19:05:29.065000 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:29.064960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:29.065354 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:29.065068 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:29.065354 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:29.065138 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:05:29.065354 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:29.065161 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf757ff8-cknd2: secret "image-registry-tls" not found Sep 16 19:05:29.065354 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:29.065224 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls podName:4162c94c-63b2-42a9-9906-869b44645018 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:33.065204717 +0000 UTC m=+41.614261976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls") pod "image-registry-75bf757ff8-cknd2" (UID: "4162c94c-63b2-42a9-9906-869b44645018") : secret "image-registry-tls" not found Sep 16 19:05:29.065354 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:29.065239 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:29.065354 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:29.065292 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls podName:fb283ce1-2ad2-4616-b24f-9560df64c816 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:33.065276625 +0000 UTC m=+41.614333878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls") pod "dns-default-fjz6k" (UID: "fb283ce1-2ad2-4616-b24f-9560df64c816") : secret "dns-default-metrics-tls" not found Sep 16 19:05:30.690147 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:30.689889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:30.773576 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:30.773536 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:32.826806 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:32.826759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:32.967202 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:32.967160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:33.095698 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:33.095652 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:33.095875 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:33.095744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:33.095875 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:33.095813 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:05:33.095875 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:33.095832 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf757ff8-cknd2: secret "image-registry-tls" not found Sep 16 19:05:33.096045 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:33.095894 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:33.096045 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:33.095903 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls podName:4162c94c-63b2-42a9-9906-869b44645018 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:41.095882427 +0000 UTC m=+49.644939690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls") pod "image-registry-75bf757ff8-cknd2" (UID: "4162c94c-63b2-42a9-9906-869b44645018") : secret "image-registry-tls" not found Sep 16 19:05:33.096045 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:33.095998 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls podName:fb283ce1-2ad2-4616-b24f-9560df64c816 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:41.095973798 +0000 UTC m=+49.645031053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls") pod "dns-default-fjz6k" (UID: "fb283ce1-2ad2-4616-b24f-9560df64c816") : secret "dns-default-metrics-tls" not found Sep 16 19:05:37.170048 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.169997 2562 generic.go:356] "Generic (PLEG): container finished" podID="f01f26bd-0336-4941-bc54-62734d65f013" containerID="3d1b0bd4e08348f419ca2c1d53abbd03055af6ca9d65df9993b29ca57dcc2a22" exitCode=0 Sep 16 19:05:37.170520 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.170082 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" event={"ID":"f01f26bd-0336-4941-bc54-62734d65f013","Type":"ContainerDied","Data":"3d1b0bd4e08348f419ca2c1d53abbd03055af6ca9d65df9993b29ca57dcc2a22"} Sep 16 19:05:37.172215 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.172190 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" event={"ID":"727aee0d-4e52-4e82-a07d-f7eed7bcc118","Type":"ContainerStarted","Data":"6715c7a77e19780b4a1238b4975cff922811a811a7d0e3fe15abb3a692663f67"} Sep 16 19:05:37.172316 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.172231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" event={"ID":"727aee0d-4e52-4e82-a07d-f7eed7bcc118","Type":"ContainerStarted","Data":"7ad7c0a8b863165fdce084e5b67e854790cb8e4b41b9aeafe38e706bf282f5ab"} Sep 16 19:05:37.172316 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.172251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" event={"ID":"727aee0d-4e52-4e82-a07d-f7eed7bcc118","Type":"ContainerStarted","Data":"ffceecdbb53420193b3d4d7caa987684c6a428810e6a5885e3160e71a69d8b5f"} Sep 16 19:05:37.173675 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.173656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w5tdv" event={"ID":"2a0bfe9b-6279-4c42-9568-812c00a190e8","Type":"ContainerStarted","Data":"4f87e754c67aaae317fccb1fd8e208b1efff830179830dd0ad656b5e01f81ce7"} Sep 16 19:05:37.175216 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.175197 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" event={"ID":"be43e7da-de98-4725-9999-00f73403f065","Type":"ContainerStarted","Data":"12c7ce16caf6916b05611ee37272da06cf75ba82b732e18fec73b8815b71c13f"} Sep 16 19:05:37.176578 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.176556 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" event={"ID":"38a44740-3ffc-48fc-9895-637cf651e633","Type":"ContainerStarted","Data":"30407b10bb86f40d492cd6715a59bd57300b8337ba3756a44dc40fa5becc823d"} Sep 16 19:05:37.199620 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.199603 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:37.201450 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.201426 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:05:37.224219 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.224164 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" podStartSLOduration=24.706783268 podStartE2EDuration="35.224143957s" totalImagesPullingTime="10.517360689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:37.223177255 +0000 UTC m=+45.772234529" watchObservedRunningTime="2026-09-16 19:05:37.224143957 +0000 UTC m=+45.773201233" Sep 16 19:05:37.242513 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.242472 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w5tdv" podStartSLOduration=1.7890233389999999 podStartE2EDuration="12.242457771s" totalImagesPullingTime="10.453434432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:37.241400805 +0000 UTC m=+45.790458078" watchObservedRunningTime="2026-09-16 19:05:37.242457771 +0000 UTC m=+45.791515045" Sep 16 19:05:37.266409 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.266356 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" podStartSLOduration=24.760396345 podStartE2EDuration="35.266328848s" totalImagesPullingTime="10.505932503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:37.266187754 +0000 UTC m=+45.815245163" watchObservedRunningTime="2026-09-16 19:05:37.266328848 +0000 UTC m=+45.815386119" Sep 16 19:05:37.285731 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:37.285675 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" podStartSLOduration=24.798732717 podStartE2EDuration="35.28565389s" totalImagesPullingTime="10.486921173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:37.284431091 +0000 UTC m=+45.833488356" watchObservedRunningTime="2026-09-16 19:05:37.28565389 +0000 UTC m=+45.834711165" Sep 16 19:05:38.182687 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:38.182648 2562 generic.go:356] "Generic (PLEG): container finished" podID="f01f26bd-0336-4941-bc54-62734d65f013" containerID="382f11b94188b84851d2330442c1912ca02f50316a3bfa3c2d7a8ef2e499dd16" exitCode=0 Sep 16 19:05:38.183090 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:38.182737 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" event={"ID":"f01f26bd-0336-4941-bc54-62734d65f013","Type":"ContainerDied","Data":"382f11b94188b84851d2330442c1912ca02f50316a3bfa3c2d7a8ef2e499dd16"} Sep 16 19:05:38.605482 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:38.605449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:38.727425 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:38.727394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:38.858906 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:38.858815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:39.193066 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:39.192960 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" event={"ID":"f01f26bd-0336-4941-bc54-62734d65f013","Type":"ContainerStarted","Data":"2ed3f9fbc027e42a1d36423f958bbd2eb95f262da002322c0657c55a7046eeb1"} Sep 16 19:05:39.227031 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:39.226952 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r2hzb" podStartSLOduration=7.920169261 podStartE2EDuration="47.226936342s" totalImagesPullingTime="39.306767081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:39.226045668 +0000 UTC m=+47.775102941" watchObservedRunningTime="2026-09-16 19:05:39.226936342 +0000 UTC m=+47.775993629" Sep 16 19:05:39.491496 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:39.491406 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:39.594296 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:39.594262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:39.697046 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:39.697001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:39.801221 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:39.801113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:39.973080 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:39.973045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:40.075735 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:40.075645 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:40.563626 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:40.563583 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:41.161218 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:41.161190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:41.161384 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:41.161242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:41.161384 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:41.161333 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:41.161384 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:41.161343 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:05:41.161384 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:41.161362 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf757ff8-cknd2: secret "image-registry-tls" not found Sep 16 19:05:41.161384 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:41.161385 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls podName:fb283ce1-2ad2-4616-b24f-9560df64c816 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:57.161371443 +0000 UTC m=+65.710428695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls") pod "dns-default-fjz6k" (UID: "fb283ce1-2ad2-4616-b24f-9560df64c816") : secret "dns-default-metrics-tls" not found Sep 16 19:05:41.161544 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:41.161403 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls podName:4162c94c-63b2-42a9-9906-869b44645018 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:57.161390885 +0000 UTC m=+65.710448137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls") pod "image-registry-75bf757ff8-cknd2" (UID: "4162c94c-63b2-42a9-9906-869b44645018") : secret "image-registry-tls" not found Sep 16 19:05:45.698568 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:45.698504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:45.816236 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:45.816198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:45.910881 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:45.910844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:46.009812 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:46.009726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:46.119490 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:46.119434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:46.247830 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:46.247800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:47.688604 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:47.688565 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:50.835088 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:50.835054 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:50.917714 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:50.917657 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:53.125583 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:53.125555 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7hqb" Sep 16 19:05:53.347409 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:53.347364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:53.490609 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:53.490512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:54.621482 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:54.621448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:54.706502 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:54.706472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:54.784443 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:54.784415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:55.736998 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:55.736966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:55.837350 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:55.837316 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:57.179421 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.179386 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:05:57.179897 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.179447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:05:57.179897 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:57.179544 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:57.179897 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:57.179548 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:05:57.179897 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:57.179568 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf757ff8-cknd2: secret "image-registry-tls" not found Sep 16 19:05:57.179897 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:57.179596 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls podName:fb283ce1-2ad2-4616-b24f-9560df64c816 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:29.179581589 +0000 UTC m=+97.728638840 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls") pod "dns-default-fjz6k" (UID: "fb283ce1-2ad2-4616-b24f-9560df64c816") : secret "dns-default-metrics-tls" not found Sep 16 19:05:57.179897 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:57.179610 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls podName:4162c94c-63b2-42a9-9906-869b44645018 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:29.179604788 +0000 UTC m=+97.728662039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls") pod "image-registry-75bf757ff8-cknd2" (UID: "4162c94c-63b2-42a9-9906-869b44645018") : secret "image-registry-tls" not found Sep 16 19:05:57.480789 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.480705 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:57.585626 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.585591 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:57.684205 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.684173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:57.684363 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.684236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:05:57.687153 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.687132 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 19:05:57.687326 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.687155 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 19:05:57.695412 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:57.695392 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 19:05:57.695492 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:05:57.695443 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs podName:10b511a5-508f-4ce7-988a-7abc0d8e70da nodeName:}" failed. No retries permitted until 2026-09-16 19:07:01.695428635 +0000 UTC m=+130.244485891 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs") pod "network-metrics-daemon-mzvlv" (UID: "10b511a5-508f-4ce7-988a-7abc0d8e70da") : secret "metrics-daemon-secret" not found Sep 16 19:05:57.698151 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.698135 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 19:05:57.708635 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.708613 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpjx8\" (UniqueName: \"kubernetes.io/projected/49609a4b-5c7b-412e-b0ae-01d78c0ea546-kube-api-access-dpjx8\") pod \"network-check-target-cnrd5\" (UID: \"49609a4b-5c7b-412e-b0ae-01d78c0ea546\") " pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:57.785199 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.785110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:57.789191 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.789166 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 19:05:57.798193 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.798170 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a31ef62-788d-40d4-b735-08587d7e341d-original-pull-secret\") pod \"global-pull-secret-syncer-hjmsb\" (UID: \"8a31ef62-788d-40d4-b735-08587d7e341d\") " pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:57.877186 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.877135 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjmsb" Sep 16 19:05:57.890876 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.890857 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45r44\"" Sep 16 19:05:57.899232 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:57.899213 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:05:58.156481 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:58.156449 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hjmsb"] Sep 16 19:05:58.159469 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:05:58.159444 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a31ef62_788d_40d4_b735_08587d7e341d.slice/crio-4d709cea0311fa05caa0881d786da29b29d378af8dc459ad89c45d1cd1f9b646 WatchSource:0}: Error finding container 4d709cea0311fa05caa0881d786da29b29d378af8dc459ad89c45d1cd1f9b646: Status 404 returned error can't find the container with id 4d709cea0311fa05caa0881d786da29b29d378af8dc459ad89c45d1cd1f9b646 Sep 16 19:05:58.193952 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:58.193925 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cnrd5"] Sep 16 19:05:58.196876 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:05:58.196849 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49609a4b_5c7b_412e_b0ae_01d78c0ea546.slice/crio-388bfb0fcea642f45053a7ef724ca8e2775ce31e607775b9e11b0b5aa81f2b24 WatchSource:0}: Error finding container 388bfb0fcea642f45053a7ef724ca8e2775ce31e607775b9e11b0b5aa81f2b24: Status 404 returned error can't find the container with id 388bfb0fcea642f45053a7ef724ca8e2775ce31e607775b9e11b0b5aa81f2b24 Sep 16 19:05:58.249411 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:58.249372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hjmsb" event={"ID":"8a31ef62-788d-40d4-b735-08587d7e341d","Type":"ContainerStarted","Data":"4d709cea0311fa05caa0881d786da29b29d378af8dc459ad89c45d1cd1f9b646"} Sep 16 19:05:58.250470 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:58.250449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cnrd5" event={"ID":"49609a4b-5c7b-412e-b0ae-01d78c0ea546","Type":"ContainerStarted","Data":"388bfb0fcea642f45053a7ef724ca8e2775ce31e607775b9e11b0b5aa81f2b24"} Sep 16 19:05:59.850880 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:59.850808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:59.952754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:05:59.952640 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:00.035512 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:00.035470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:02.485915 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:02.485874 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:02.564665 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:02.564575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:04.274318 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:04.274279 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hjmsb" event={"ID":"8a31ef62-788d-40d4-b735-08587d7e341d","Type":"ContainerStarted","Data":"566a935c24929ab364490c7f576a39aee081292730c7f60ecc9bcab29b0e7be9"} Sep 16 19:06:04.276059 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:04.276024 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cnrd5" event={"ID":"49609a4b-5c7b-412e-b0ae-01d78c0ea546","Type":"ContainerStarted","Data":"7f1abc8aa1a32099a2fd569184eed4f2acccab651c06b3ff7ce18c5f063df41c"} Sep 16 19:06:04.276183 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:04.276153 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:06:04.295651 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:04.295613 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hjmsb" podStartSLOduration=66.687822907 podStartE2EDuration="1m12.295600548s" totalImagesPullingTime="5.607777641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:04.294359737 +0000 UTC m=+72.843417010" watchObservedRunningTime="2026-09-16 19:06:04.295600548 +0000 UTC m=+72.844657821" Sep 16 19:06:15.565656 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:15.565533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:16.907790 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:16.907751 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:21.797587 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:21.797552 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:21.877666 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:21.877628 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:21.953834 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:21.953796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:29.208899 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:29.208843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:06:29.208899 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:29.208916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:06:29.209479 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:29.209045 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:06:29.209479 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:29.209059 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:06:29.209479 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:29.209077 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf757ff8-cknd2: secret "image-registry-tls" not found Sep 16 19:06:29.209479 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:29.209135 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls podName:fb283ce1-2ad2-4616-b24f-9560df64c816 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:33.209112487 +0000 UTC m=+161.758169743 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls") pod "dns-default-fjz6k" (UID: "fb283ce1-2ad2-4616-b24f-9560df64c816") : secret "dns-default-metrics-tls" not found Sep 16 19:06:29.209479 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:29.209155 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls podName:4162c94c-63b2-42a9-9906-869b44645018 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:33.209145741 +0000 UTC m=+161.758202996 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls") pod "image-registry-75bf757ff8-cknd2" (UID: "4162c94c-63b2-42a9-9906-869b44645018") : secret "image-registry-tls" not found Sep 16 19:06:35.282432 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:35.282399 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cnrd5" Sep 16 19:06:35.298755 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:35.298705 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cnrd5" podStartSLOduration=97.73892726 podStartE2EDuration="1m43.298691671s" totalImagesPullingTime="5.559764411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:04.318165811 +0000 UTC m=+72.867223084" watchObservedRunningTime="2026-09-16 19:06:35.298691671 +0000 UTC m=+103.847748989" Sep 16 19:06:42.423571 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:42.423538 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:42.518996 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:42.518952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:44.725320 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:44.725286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:44.825944 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:44.825914 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:45.788581 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.788547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:45.795842 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.795813 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f6sfj"] Sep 16 19:06:45.798835 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.798816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:45.802894 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.802879 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 19:06:45.802987 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.802879 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 19:06:45.803465 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.803448 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 19:06:45.803550 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.803448 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 19:06:45.804027 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.804001 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-27w29\"" Sep 16 19:06:45.817346 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.817321 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f6sfj"] Sep 16 19:06:45.899770 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.899732 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:45.932652 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.932615 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94ea5312-94ba-403e-88cf-ad816a2daad9-crio-socket\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:45.932652 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.932657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:45.932811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.932731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:45.932811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.932764 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d8d\" (UniqueName: \"kubernetes.io/projected/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-api-access-f9d8d\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:45.932911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:45.932861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ea5312-94ba-403e-88cf-ad816a2daad9-data-volume\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.034160 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.034121 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ea5312-94ba-403e-88cf-ad816a2daad9-data-volume\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.034349 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.034170 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94ea5312-94ba-403e-88cf-ad816a2daad9-crio-socket\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.034349 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.034193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.034349 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.034271 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.034349 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.034317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d8d\" (UniqueName: \"kubernetes.io/projected/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-api-access-f9d8d\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.034349 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:46.034325 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:46.034587 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:46.034359 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:46.034587 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.034268 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94ea5312-94ba-403e-88cf-ad816a2daad9-crio-socket\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.034587 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:46.034411 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:46.534395398 +0000 UTC m=+115.083452650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : configmap "kube-rbac-proxy" not found Sep 16 19:06:46.034587 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:46.034426 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:46.534419605 +0000 UTC m=+115.083476856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:46.034587 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.034453 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ea5312-94ba-403e-88cf-ad816a2daad9-data-volume\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.043362 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.043306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d8d\" (UniqueName: \"kubernetes.io/projected/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-api-access-f9d8d\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.537856 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.537803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.538175 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:46.537897 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:46.538175 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:46.537926 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:46.538175 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:46.537998 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:47.537983654 +0000 UTC m=+116.087040908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : configmap "kube-rbac-proxy" not found Sep 16 19:06:46.538175 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:46.538031 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:46.538175 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:46.538098 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:47.538082185 +0000 UTC m=+116.087139437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:47.120870 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.120837 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s"] Sep 16 19:06:47.123751 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.123735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s" Sep 16 19:06:47.126428 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.126411 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-x28d6\"" Sep 16 19:06:47.129182 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.129156 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s"] Sep 16 19:06:47.244772 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.244736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bcb\" (UniqueName: \"kubernetes.io/projected/96bdd295-a0ef-424a-8d86-9b812e6c77ea-kube-api-access-s7bcb\") pod \"network-check-source-6f8b768455-sgz7s\" (UID: \"96bdd295-a0ef-424a-8d86-9b812e6c77ea\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s" Sep 16 19:06:47.345753 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.345716 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bcb\" (UniqueName: \"kubernetes.io/projected/96bdd295-a0ef-424a-8d86-9b812e6c77ea-kube-api-access-s7bcb\") pod \"network-check-source-6f8b768455-sgz7s\" (UID: \"96bdd295-a0ef-424a-8d86-9b812e6c77ea\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s" Sep 16 19:06:47.354307 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.354285 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bcb\" (UniqueName: \"kubernetes.io/projected/96bdd295-a0ef-424a-8d86-9b812e6c77ea-kube-api-access-s7bcb\") pod \"network-check-source-6f8b768455-sgz7s\" (UID: \"96bdd295-a0ef-424a-8d86-9b812e6c77ea\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s" Sep 16 19:06:47.433155 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.433072 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s" Sep 16 19:06:47.547247 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.547215 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:47.547428 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.547281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:47.547428 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:47.547367 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:47.547428 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:47.547369 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:47.547536 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:47.547432 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:49.54740845 +0000 UTC m=+118.096465702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : configmap "kube-rbac-proxy" not found Sep 16 19:06:47.547536 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:47.547449 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:49.547441818 +0000 UTC m=+118.096499071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:47.700813 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.700742 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s"] Sep 16 19:06:47.705124 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:06:47.705089 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bdd295_a0ef_424a_8d86_9b812e6c77ea.slice/crio-5fa9e2064473d8a3fc9d3496729c4ebfb547367798c493e0a2d431b70c8410d6 WatchSource:0}: Error finding container 5fa9e2064473d8a3fc9d3496729c4ebfb547367798c493e0a2d431b70c8410d6: Status 404 returned error can't find the container with id 5fa9e2064473d8a3fc9d3496729c4ebfb547367798c493e0a2d431b70c8410d6 Sep 16 19:06:47.909560 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:47.909531 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vq45m_5bf475c5-b68d-4235-b53e-ff2420b1e556/dns-node-resolver/0.log" Sep 16 19:06:48.410145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:48.410113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s" event={"ID":"96bdd295-a0ef-424a-8d86-9b812e6c77ea","Type":"ContainerStarted","Data":"aa628efbcd9bba74756aa31deb7cd258422a18176bd0e059d2a502a326833162"} Sep 16 19:06:48.410608 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:48.410157 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s" event={"ID":"96bdd295-a0ef-424a-8d86-9b812e6c77ea","Type":"ContainerStarted","Data":"5fa9e2064473d8a3fc9d3496729c4ebfb547367798c493e0a2d431b70c8410d6"} Sep 16 19:06:48.440952 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:48.440902 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sgz7s" podStartSLOduration=1.440885177 podStartE2EDuration="1.440885177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:48.435331004 +0000 UTC m=+116.984388278" watchObservedRunningTime="2026-09-16 19:06:48.440885177 +0000 UTC m=+116.989942451" Sep 16 19:06:48.475233 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:48.475196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:48.583458 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:48.583419 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:48.687511 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:48.687431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:48.711841 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:48.711818 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ft7jx_9f54592e-2056-4dd4-969c-eaeec3227210/node-ca/0.log" Sep 16 19:06:49.510453 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:49.510427 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w5tdv_2a0bfe9b-6279-4c42-9568-812c00a190e8/serve-healthcheck-canary/0.log" Sep 16 19:06:49.563283 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:49.563255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:49.563386 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:49.563337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:49.563432 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:49.563415 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:49.563481 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:49.563464 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:49.563520 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:49.563494 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:53.563474899 +0000 UTC m=+122.112532169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:49.563520 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:49.563517 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:53.563504512 +0000 UTC m=+122.112561764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : configmap "kube-rbac-proxy" not found Sep 16 19:06:51.189687 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:51.189644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:51.280603 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:51.280560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:51.622005 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:51.621975 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:51.709325 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:51.709289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:51.911940 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:51.911838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:52.060995 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:52.060959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:53.568950 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:53.568912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:53.599674 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:53.599643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:53.599806 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:53.599707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:06:53.599806 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:53.599778 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:53.599806 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:53.599785 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:53.599922 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:53.599833 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:01.599817875 +0000 UTC m=+130.148875127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : configmap "kube-rbac-proxy" not found Sep 16 19:06:53.599922 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:06:53.599845 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:01.599839289 +0000 UTC m=+130.148896540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:53.677353 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:53.677320 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:55.148423 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:55.148387 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:57.891798 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:57.891762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:58.020255 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:58.020226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:58.141142 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:58.141108 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:58.264270 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:58.264185 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:58.368311 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:58.368280 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:58.538485 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:58.538406 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:59.095511 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:59.095476 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:59.336600 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:59.336550 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:59.534852 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:59.534761 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:59.980333 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:06:59.980301 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:00.195530 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:00.195499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:00.374406 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:00.374365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:00.753637 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:00.753553 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:01.436605 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:01.436568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:01.572792 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:01.572755 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:01.664507 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:01.664474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:07:01.664707 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:01.664559 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:07:01.664707 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:07:01.664626 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:07:01.664707 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:07:01.664706 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls podName:94ea5312-94ba-403e-88cf-ad816a2daad9 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:17.664691077 +0000 UTC m=+146.213748330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f6sfj" (UID: "94ea5312-94ba-403e-88cf-ad816a2daad9") : secret "insights-runtime-extractor-tls" not found Sep 16 19:07:01.665122 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:01.665102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94ea5312-94ba-403e-88cf-ad816a2daad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:07:01.740439 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:01.740355 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:01.764912 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:01.764882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:07:01.765048 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:07:01.764993 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 19:07:01.765091 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:07:01.765081 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs podName:10b511a5-508f-4ce7-988a-7abc0d8e70da nodeName:}" failed. No retries permitted until 2026-09-16 19:09:03.765063475 +0000 UTC m=+252.314120748 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs") pod "network-metrics-daemon-mzvlv" (UID: "10b511a5-508f-4ce7-988a-7abc0d8e70da") : secret "metrics-daemon-secret" not found Sep 16 19:07:01.938347 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:01.938319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:02.142454 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:02.142413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:02.508898 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:02.508809 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:02.728457 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:02.728423 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:02.820456 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:02.820375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:02.930459 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:02.930432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:03.142404 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:03.142376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:03.641763 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:03.641729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:04.144902 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:04.144870 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:04.386148 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:04.386110 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:04.559246 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:04.559160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:04.738964 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:04.738925 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:05.112485 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:05.112453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:05.641482 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:05.641449 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" podUID="727aee0d-4e52-4e82-a07d-f7eed7bcc118" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 19:07:05.728534 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:05.728501 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:06.397343 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:06.397311 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:06.511199 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:06.511160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:06.717525 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:06.717432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:07.130352 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:07.130312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:07.645709 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:07.645672 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:07.746773 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:07.746737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:08.125844 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:08.125812 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:08.318190 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:08.318157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:08.507988 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:08.507894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:15.641293 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:15.641256 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" podUID="727aee0d-4e52-4e82-a07d-f7eed7bcc118" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 19:07:16.760122 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:16.760091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:16.866034 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:16.865978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:16.960853 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:16.960818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:17.205937 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:17.205899 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:17.686198 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:17.686164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:07:17.688565 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:17.688543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94ea5312-94ba-403e-88cf-ad816a2daad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f6sfj\" (UID: \"94ea5312-94ba-403e-88cf-ad816a2daad9\") " pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:07:17.799961 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:17.799926 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:17.900049 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:17.899993 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:17.909404 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:17.909381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-27w29\"" Sep 16 19:07:17.917069 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:17.917047 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f6sfj" Sep 16 19:07:18.190992 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:18.190961 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f6sfj"] Sep 16 19:07:18.194182 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:07:18.194156 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ea5312_94ba_403e_88cf_ad816a2daad9.slice/crio-629f6935dd9b54e36034a0bdf681733fa2f9cdc67a05d905df54b88a96e2e03e WatchSource:0}: Error finding container 629f6935dd9b54e36034a0bdf681733fa2f9cdc67a05d905df54b88a96e2e03e: Status 404 returned error can't find the container with id 629f6935dd9b54e36034a0bdf681733fa2f9cdc67a05d905df54b88a96e2e03e Sep 16 19:07:18.498372 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:18.498294 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f6sfj" event={"ID":"94ea5312-94ba-403e-88cf-ad816a2daad9","Type":"ContainerStarted","Data":"f113cd5b3fdec65fb06503f8d1fd8b227e35e74b3c78674f97c5f95c76ea5a2c"} Sep 16 19:07:18.498372 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:18.498349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f6sfj" event={"ID":"94ea5312-94ba-403e-88cf-ad816a2daad9","Type":"ContainerStarted","Data":"629f6935dd9b54e36034a0bdf681733fa2f9cdc67a05d905df54b88a96e2e03e"} Sep 16 19:07:18.871249 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:18.871214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:19.209118 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:19.209081 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:19.502734 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:19.502651 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f6sfj" event={"ID":"94ea5312-94ba-403e-88cf-ad816a2daad9","Type":"ContainerStarted","Data":"31aa20fbabc464aabc1d4f62088cbf740912c9bc22eb58a3412760b17565cc60"} Sep 16 19:07:21.211950 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:21.211915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:21.312636 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:21.312604 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:21.511888 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:21.511805 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f6sfj" event={"ID":"94ea5312-94ba-403e-88cf-ad816a2daad9","Type":"ContainerStarted","Data":"8336abf7ca1472161ce2c276df3291524287d4aff9b5974b6eca122ad32650ac"} Sep 16 19:07:21.534450 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:21.534401 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f6sfj" podStartSLOduration=34.120801436 podStartE2EDuration="36.534384898s" totalImagesPullingTime="2.413583462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:06:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:21.532307261 +0000 UTC m=+150.081364535" watchObservedRunningTime="2026-09-16 19:07:21.534384898 +0000 UTC m=+150.083442172" Sep 16 19:07:23.041302 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:23.041263 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:23.217475 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:23.217438 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:23.478874 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:23.478840 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:25.562810 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:25.562770 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:25.641337 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:25.641302 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" podUID="727aee0d-4e52-4e82-a07d-f7eed7bcc118" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 19:07:25.641481 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:25.641365 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" Sep 16 19:07:25.641836 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:25.641807 2562 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6715c7a77e19780b4a1238b4975cff922811a811a7d0e3fe15abb3a692663f67"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 16 19:07:25.641876 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:25.641857 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" podUID="727aee0d-4e52-4e82-a07d-f7eed7bcc118" containerName="service-proxy" containerID="cri-o://6715c7a77e19780b4a1238b4975cff922811a811a7d0e3fe15abb3a692663f67" gracePeriod=30 Sep 16 19:07:26.529408 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:26.529374 2562 generic.go:356] "Generic (PLEG): container finished" podID="727aee0d-4e52-4e82-a07d-f7eed7bcc118" containerID="6715c7a77e19780b4a1238b4975cff922811a811a7d0e3fe15abb3a692663f67" exitCode=2 Sep 16 19:07:26.529574 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:26.529417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" event={"ID":"727aee0d-4e52-4e82-a07d-f7eed7bcc118","Type":"ContainerDied","Data":"6715c7a77e19780b4a1238b4975cff922811a811a7d0e3fe15abb3a692663f67"} Sep 16 19:07:26.529574 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:26.529449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586554554b-4qmzb" event={"ID":"727aee0d-4e52-4e82-a07d-f7eed7bcc118","Type":"ContainerStarted","Data":"006897dca8aa9144cfb0bad8401ac9f7546727bda54d021a45c87c7a079e033a"} Sep 16 19:07:27.351615 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.351583 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:27.565632 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.565597 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:27.687062 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.686971 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ff7n8"] Sep 16 19:07:27.690062 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.690045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.693348 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.693323 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 19:07:27.693509 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.693372 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 19:07:27.693591 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.693568 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 19:07:27.693634 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.693588 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 19:07:27.694156 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.694140 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6rp57\"" Sep 16 19:07:27.694820 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.694805 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 19:07:27.695108 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.695094 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 19:07:27.720912 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.720886 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:27.764758 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.764728 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cccb3134-b55b-44de-8f31-6f7182e05922-root\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.764911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.764769 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccb3134-b55b-44de-8f31-6f7182e05922-metrics-client-ca\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.764911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.764807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.764911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.764830 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-accelerators-collector-config\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.764911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.764847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-textfile\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.764911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.764862 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-wtmp\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.764911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.764879 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-tls\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.765132 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.764937 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc58m\" (UniqueName: \"kubernetes.io/projected/cccb3134-b55b-44de-8f31-6f7182e05922-kube-api-access-gc58m\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.765132 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.765006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cccb3134-b55b-44de-8f31-6f7182e05922-sys\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866093 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cccb3134-b55b-44de-8f31-6f7182e05922-sys\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cccb3134-b55b-44de-8f31-6f7182e05922-root\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccb3134-b55b-44de-8f31-6f7182e05922-metrics-client-ca\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866178 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cccb3134-b55b-44de-8f31-6f7182e05922-sys\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-accelerators-collector-config\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-textfile\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-wtmp\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866548 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866191 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cccb3134-b55b-44de-8f31-6f7182e05922-root\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866548 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-tls\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866548 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc58m\" (UniqueName: \"kubernetes.io/projected/cccb3134-b55b-44de-8f31-6f7182e05922-kube-api-access-gc58m\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866548 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866390 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-wtmp\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866548 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:07:27.866472 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 19:07:27.866548 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:07:27.866549 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-tls podName:cccb3134-b55b-44de-8f31-6f7182e05922 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:28.366527717 +0000 UTC m=+156.915584972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-tls") pod "node-exporter-ff7n8" (UID: "cccb3134-b55b-44de-8f31-6f7182e05922") : secret "node-exporter-tls" not found Sep 16 19:07:27.866802 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-textfile\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866802 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866789 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-accelerators-collector-config\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.866891 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.866839 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccb3134-b55b-44de-8f31-6f7182e05922-metrics-client-ca\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.868445 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.868429 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.880218 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.880198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc58m\" (UniqueName: \"kubernetes.io/projected/cccb3134-b55b-44de-8f31-6f7182e05922-kube-api-access-gc58m\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:27.890159 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:27.890133 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:28.028857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.028775 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:28.151821 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.151789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:28.276142 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.276113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:28.346757 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:07:28.346717 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" podUID="4162c94c-63b2-42a9-9906-869b44645018" Sep 16 19:07:28.369994 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.369970 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-tls\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:28.372136 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.372117 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cccb3134-b55b-44de-8f31-6f7182e05922-node-exporter-tls\") pod \"node-exporter-ff7n8\" (UID: \"cccb3134-b55b-44de-8f31-6f7182e05922\") " pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:28.377175 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:07:28.377152 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fjz6k" podUID="fb283ce1-2ad2-4616-b24f-9560df64c816" Sep 16 19:07:28.536161 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.536131 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:07:28.536161 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.536159 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fjz6k" Sep 16 19:07:28.599114 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.599006 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ff7n8" Sep 16 19:07:28.609848 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:07:28.609817 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccb3134_b55b_44de_8f31_6f7182e05922.slice/crio-d17d4eed4c2f2183889b59244695670a2483738bfa3c5ef676bd2c64659b89a8 WatchSource:0}: Error finding container d17d4eed4c2f2183889b59244695670a2483738bfa3c5ef676bd2c64659b89a8: Status 404 returned error can't find the container with id d17d4eed4c2f2183889b59244695670a2483738bfa3c5ef676bd2c64659b89a8 Sep 16 19:07:28.620952 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.620923 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:28.737530 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.737496 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:28.855477 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:28.855364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:29.541586 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:29.541545 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ff7n8" event={"ID":"cccb3134-b55b-44de-8f31-6f7182e05922","Type":"ContainerStarted","Data":"d17d4eed4c2f2183889b59244695670a2483738bfa3c5ef676bd2c64659b89a8"} Sep 16 19:07:29.995304 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:07:29.995262 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mzvlv" podUID="10b511a5-508f-4ce7-988a-7abc0d8e70da" Sep 16 19:07:30.546378 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:30.546280 2562 generic.go:356] "Generic (PLEG): container finished" podID="cccb3134-b55b-44de-8f31-6f7182e05922" containerID="5078fc8d0b46d7b8d90e223334f82f3ed501966936d21577b122cb3386a6ad45" exitCode=0 Sep 16 19:07:30.546378 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:30.546324 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ff7n8" event={"ID":"cccb3134-b55b-44de-8f31-6f7182e05922","Type":"ContainerDied","Data":"5078fc8d0b46d7b8d90e223334f82f3ed501966936d21577b122cb3386a6ad45"} Sep 16 19:07:30.812179 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:30.812088 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:30.940956 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:30.940924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:31.055804 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:31.055770 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:31.453680 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:31.453642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:31.551943 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:31.551914 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ff7n8" event={"ID":"cccb3134-b55b-44de-8f31-6f7182e05922","Type":"ContainerStarted","Data":"f2db31972ad9c5f32fc16a12060fa833de72fc2e8047867e91a3e9306d9395cc"} Sep 16 19:07:31.552427 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:31.551963 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ff7n8" event={"ID":"cccb3134-b55b-44de-8f31-6f7182e05922","Type":"ContainerStarted","Data":"0f771e3ceff31fba72f5c51d9d049f75b0c0e9bebbf6ab2bc7881c86ba0cac24"} Sep 16 19:07:31.592175 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:31.592119 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ff7n8" podStartSLOduration=3.432657412 podStartE2EDuration="4.592100737s" totalImagesPullingTime="1.098836734s" totalInitContainerRuntime="60.606591ms" isStatefulPod=true podCreationTimestamp="2026-09-16 19:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:31.591666549 +0000 UTC m=+160.140723825" watchObservedRunningTime="2026-09-16 19:07:31.592100737 +0000 UTC m=+160.141158012" Sep 16 19:07:32.323224 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:32.323189 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:32.510288 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:32.510256 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:33.309979 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.309934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:07:33.310481 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.309990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:07:33.312303 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.312275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb283ce1-2ad2-4616-b24f-9560df64c816-metrics-tls\") pod \"dns-default-fjz6k\" (UID: \"fb283ce1-2ad2-4616-b24f-9560df64c816\") " pod="openshift-dns/dns-default-fjz6k" Sep 16 19:07:33.312426 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.312409 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"image-registry-75bf757ff8-cknd2\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:07:33.342071 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.342049 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pps2s\"" Sep 16 19:07:33.342288 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.342267 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pvzq\"" Sep 16 19:07:33.347714 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.347696 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:07:33.347811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.347716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fjz6k" Sep 16 19:07:33.641445 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.641406 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75bf757ff8-cknd2"] Sep 16 19:07:33.643406 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:07:33.643379 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4162c94c_63b2_42a9_9906_869b44645018.slice/crio-4122c54227af712fe93b94e86eaf265f6ce7e72507abdea00b51d86da052c703 WatchSource:0}: Error finding container 4122c54227af712fe93b94e86eaf265f6ce7e72507abdea00b51d86da052c703: Status 404 returned error can't find the container with id 4122c54227af712fe93b94e86eaf265f6ce7e72507abdea00b51d86da052c703 Sep 16 19:07:33.648643 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:33.648621 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fjz6k"] Sep 16 19:07:33.653098 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:07:33.653071 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb283ce1_2ad2_4616_b24f_9560df64c816.slice/crio-22ccb9c6f2d65f11cd4a7ea6c8dbd60eab5caf2ab517ebf0d57061d11cc8a880 WatchSource:0}: Error finding container 22ccb9c6f2d65f11cd4a7ea6c8dbd60eab5caf2ab517ebf0d57061d11cc8a880: Status 404 returned error can't find the container with id 22ccb9c6f2d65f11cd4a7ea6c8dbd60eab5caf2ab517ebf0d57061d11cc8a880 Sep 16 19:07:34.322921 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:34.322891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:34.563136 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:34.563100 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" event={"ID":"4162c94c-63b2-42a9-9906-869b44645018","Type":"ContainerStarted","Data":"62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188"} Sep 16 19:07:34.563325 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:34.563157 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" event={"ID":"4162c94c-63b2-42a9-9906-869b44645018","Type":"ContainerStarted","Data":"4122c54227af712fe93b94e86eaf265f6ce7e72507abdea00b51d86da052c703"} Sep 16 19:07:34.563325 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:34.563204 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:07:34.564383 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:34.564359 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fjz6k" event={"ID":"fb283ce1-2ad2-4616-b24f-9560df64c816","Type":"ContainerStarted","Data":"22ccb9c6f2d65f11cd4a7ea6c8dbd60eab5caf2ab517ebf0d57061d11cc8a880"} Sep 16 19:07:35.320937 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:35.320907 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:35.417540 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:35.417505 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:35.570007 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:35.569916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fjz6k" event={"ID":"fb283ce1-2ad2-4616-b24f-9560df64c816","Type":"ContainerStarted","Data":"ed0477d7e6b3fa537ea90334a45018020a19e7f2a0374f8a781f649c89e22826"} Sep 16 19:07:35.570007 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:35.569993 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fjz6k" Sep 16 19:07:35.570197 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:35.570032 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fjz6k" event={"ID":"fb283ce1-2ad2-4616-b24f-9560df64c816","Type":"ContainerStarted","Data":"ea8a9afad310751e8ee7d38fd48c52ed358c6e23a70a37f72f9b9c869a594a80"} Sep 16 19:07:35.590960 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:35.590910 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" podStartSLOduration=163.590893983 podStartE2EDuration="2m43.590893983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:34.591628448 +0000 UTC m=+163.140685724" watchObservedRunningTime="2026-09-16 19:07:35.590893983 +0000 UTC m=+164.139951267" Sep 16 19:07:35.591242 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:35.591221 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fjz6k" podStartSLOduration=128.977783951 podStartE2EDuration="2m10.591213102s" totalImagesPullingTime="1.613429151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:35.589756764 +0000 UTC m=+164.138814038" watchObservedRunningTime="2026-09-16 19:07:35.591213102 +0000 UTC m=+164.140270376" Sep 16 19:07:37.200813 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:37.200771 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" podUID="38a44740-3ffc-48fc-9895-637cf651e633" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.6:8000/readyz\": dial tcp 10.132.0.6:8000: connect: connection refused" Sep 16 19:07:37.577755 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:37.577674 2562 generic.go:356] "Generic (PLEG): container finished" podID="be43e7da-de98-4725-9999-00f73403f065" containerID="12c7ce16caf6916b05611ee37272da06cf75ba82b732e18fec73b8815b71c13f" exitCode=255 Sep 16 19:07:37.577887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:37.577748 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" event={"ID":"be43e7da-de98-4725-9999-00f73403f065","Type":"ContainerDied","Data":"12c7ce16caf6916b05611ee37272da06cf75ba82b732e18fec73b8815b71c13f"} Sep 16 19:07:37.578107 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:37.578080 2562 scope.go:117] "RemoveContainer" containerID="12c7ce16caf6916b05611ee37272da06cf75ba82b732e18fec73b8815b71c13f" Sep 16 19:07:37.579368 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:37.579348 2562 generic.go:356] "Generic (PLEG): container finished" podID="38a44740-3ffc-48fc-9895-637cf651e633" containerID="30407b10bb86f40d492cd6715a59bd57300b8337ba3756a44dc40fa5becc823d" exitCode=1 Sep 16 19:07:37.579432 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:37.579413 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" event={"ID":"38a44740-3ffc-48fc-9895-637cf651e633","Type":"ContainerDied","Data":"30407b10bb86f40d492cd6715a59bd57300b8337ba3756a44dc40fa5becc823d"} Sep 16 19:07:37.579697 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:37.579686 2562 scope.go:117] "RemoveContainer" containerID="30407b10bb86f40d492cd6715a59bd57300b8337ba3756a44dc40fa5becc823d" Sep 16 19:07:38.586710 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:38.586670 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67774866f8-fhc62" event={"ID":"be43e7da-de98-4725-9999-00f73403f065","Type":"ContainerStarted","Data":"996774910dc373cc683b25bacee533e1baa4ca6d2b4c02e184c40632811ef999"} Sep 16 19:07:38.588625 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:38.588604 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" event={"ID":"38a44740-3ffc-48fc-9895-637cf651e633","Type":"ContainerStarted","Data":"215ee1adaf33c6af0236123c59cb8214c69f6682972bf340d463c1b01c6118f0"} Sep 16 19:07:38.588821 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:38.588800 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:07:38.589417 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:38.589401 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfbd9c499-fnmsk" Sep 16 19:07:43.957807 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:43.957763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:07:44.029127 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:44.029087 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:44.133817 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:44.133763 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:44.279606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:44.279491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:44.449403 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:44.449363 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:44.551371 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:44.551289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:44.660270 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:44.660235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:44.771028 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:44.770992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:44.920561 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:44.920525 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:45.576064 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:45.576032 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fjz6k" Sep 16 19:07:45.868626 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:45.868591 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:46.062816 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:46.062777 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:46.199559 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:46.199466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:46.281458 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:46.281421 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:46.491605 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:46.491513 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:46.592439 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:46.592390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:51.053663 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:51.053625 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75bf757ff8-cknd2"] Sep 16 19:07:51.057990 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:51.057944 2562 patch_prober.go:28] interesting pod/image-registry-75bf757ff8-cknd2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 16 19:07:51.058139 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:51.058008 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" podUID="4162c94c-63b2-42a9-9906-869b44645018" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:07:51.212078 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:51.212046 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:51.309237 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:51.309155 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:51.407539 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:51.407504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:54.837258 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:54.837223 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:54.936166 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:54.936127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:57.489850 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:57.489816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:57.665997 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:57.665964 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:57.749052 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:57.748941 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:58.092290 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:58.092260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:58.163730 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:58.163694 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:59.921659 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:07:59.921625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:00.064966 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:00.064932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:01.059526 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:01.059486 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:08:01.514358 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:01.514320 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:01.618458 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:01.618420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:02.825599 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:02.825558 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:03.037610 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:03.037574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:06.509722 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:06.509681 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:06.608691 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:06.608654 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:10.122880 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:10.122848 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:10.293303 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:10.293270 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:10.755271 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:10.755234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:10.862142 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:10.862104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:12.228499 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:12.228465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:12.325646 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:12.325613 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:12.431166 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:12.431137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:12.515430 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:12.515350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:12.906262 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:12.906228 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:13.056735 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:13.056696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:16.072370 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:16.072333 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" podUID="4162c94c-63b2-42a9-9906-869b44645018" containerName="registry" containerID="cri-o://62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188" gracePeriod=30 Sep 16 19:08:17.310576 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.310552 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:08:17.343836 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.343808 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4162c94c-63b2-42a9-9906-869b44645018-ca-trust-extracted\") pod \"4162c94c-63b2-42a9-9906-869b44645018\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " Sep 16 19:08:17.343965 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.343850 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bsnl\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-kube-api-access-4bsnl\") pod \"4162c94c-63b2-42a9-9906-869b44645018\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " Sep 16 19:08:17.343965 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.343875 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-image-registry-private-configuration\") pod \"4162c94c-63b2-42a9-9906-869b44645018\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " Sep 16 19:08:17.343965 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.343901 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-registry-certificates\") pod \"4162c94c-63b2-42a9-9906-869b44645018\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " Sep 16 19:08:17.343965 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.343929 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-trusted-ca\") pod \"4162c94c-63b2-42a9-9906-869b44645018\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " Sep 16 19:08:17.344151 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.343975 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-installation-pull-secrets\") pod \"4162c94c-63b2-42a9-9906-869b44645018\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " Sep 16 19:08:17.344151 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.344002 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-bound-sa-token\") pod \"4162c94c-63b2-42a9-9906-869b44645018\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " Sep 16 19:08:17.344151 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.344060 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") pod \"4162c94c-63b2-42a9-9906-869b44645018\" (UID: \"4162c94c-63b2-42a9-9906-869b44645018\") " Sep 16 19:08:17.344626 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.344574 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4162c94c-63b2-42a9-9906-869b44645018" (UID: "4162c94c-63b2-42a9-9906-869b44645018"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:08:17.344749 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.344646 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4162c94c-63b2-42a9-9906-869b44645018" (UID: "4162c94c-63b2-42a9-9906-869b44645018"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:08:17.346727 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.346701 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-kube-api-access-4bsnl" (OuterVolumeSpecName: "kube-api-access-4bsnl") pod "4162c94c-63b2-42a9-9906-869b44645018" (UID: "4162c94c-63b2-42a9-9906-869b44645018"). InnerVolumeSpecName "kube-api-access-4bsnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:08:17.346829 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.346726 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4162c94c-63b2-42a9-9906-869b44645018" (UID: "4162c94c-63b2-42a9-9906-869b44645018"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:08:17.346829 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.346751 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4162c94c-63b2-42a9-9906-869b44645018" (UID: "4162c94c-63b2-42a9-9906-869b44645018"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:08:17.346829 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.346772 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4162c94c-63b2-42a9-9906-869b44645018" (UID: "4162c94c-63b2-42a9-9906-869b44645018"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:08:17.346940 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.346884 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4162c94c-63b2-42a9-9906-869b44645018" (UID: "4162c94c-63b2-42a9-9906-869b44645018"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:08:17.354369 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.354343 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4162c94c-63b2-42a9-9906-869b44645018-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4162c94c-63b2-42a9-9906-869b44645018" (UID: "4162c94c-63b2-42a9-9906-869b44645018"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:08:17.445660 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.445558 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-installation-pull-secrets\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:08:17.445660 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.445601 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-bound-sa-token\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:08:17.445660 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.445611 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-registry-tls\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:08:17.445660 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.445619 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4162c94c-63b2-42a9-9906-869b44645018-ca-trust-extracted\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:08:17.445660 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.445627 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bsnl\" (UniqueName: \"kubernetes.io/projected/4162c94c-63b2-42a9-9906-869b44645018-kube-api-access-4bsnl\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:08:17.445660 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.445636 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4162c94c-63b2-42a9-9906-869b44645018-image-registry-private-configuration\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:08:17.445660 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.445644 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-registry-certificates\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:08:17.445660 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.445653 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4162c94c-63b2-42a9-9906-869b44645018-trusted-ca\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:08:17.722508 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.722428 2562 generic.go:356] "Generic (PLEG): container finished" podID="4162c94c-63b2-42a9-9906-869b44645018" containerID="62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188" exitCode=0 Sep 16 19:08:17.722508 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.722470 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" Sep 16 19:08:17.722508 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.722500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" event={"ID":"4162c94c-63b2-42a9-9906-869b44645018","Type":"ContainerDied","Data":"62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188"} Sep 16 19:08:17.722730 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.722538 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bf757ff8-cknd2" event={"ID":"4162c94c-63b2-42a9-9906-869b44645018","Type":"ContainerDied","Data":"4122c54227af712fe93b94e86eaf265f6ce7e72507abdea00b51d86da052c703"} Sep 16 19:08:17.722730 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.722559 2562 scope.go:117] "RemoveContainer" containerID="62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188" Sep 16 19:08:17.730845 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.730828 2562 scope.go:117] "RemoveContainer" containerID="62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188" Sep 16 19:08:17.731135 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:08:17.731113 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188\": container with ID starting with 62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188 not found: ID does not exist" containerID="62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188" Sep 16 19:08:17.731191 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.731143 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188"} err="failed to get container status \"62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188\": rpc error: code = NotFound desc = could not find container \"62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188\": container with ID starting with 62f924589fcb580bf43f8e3714692a73e0aa826fbf64d712dab5bf1f09714188 not found: ID does not exist" Sep 16 19:08:17.744162 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.744139 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75bf757ff8-cknd2"] Sep 16 19:08:17.751556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.751533 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75bf757ff8-cknd2"] Sep 16 19:08:17.804823 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.804791 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:17.906429 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.906390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:17.965467 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:17.965433 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4162c94c-63b2-42a9-9906-869b44645018" path="/var/lib/kubelet/pods/4162c94c-63b2-42a9-9906-869b44645018/volumes" Sep 16 19:08:27.494966 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:27.494932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:27.596637 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:27.596592 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:32.868482 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:32.868445 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:33.046117 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:33.046063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:47.344438 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:47.344399 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:47.876182 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:47.876156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:56.393853 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:56.393811 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:56.499490 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:08:56.499454 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:03.772609 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:03.772572 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:09:03.774800 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:03.774779 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b511a5-508f-4ce7-988a-7abc0d8e70da-metrics-certs\") pod \"network-metrics-daemon-mzvlv\" (UID: \"10b511a5-508f-4ce7-988a-7abc0d8e70da\") " pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:09:04.061880 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:04.061807 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjb6w\"" Sep 16 19:09:04.069398 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:04.069376 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzvlv" Sep 16 19:09:04.346910 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:04.346884 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mzvlv"] Sep 16 19:09:04.349386 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:09:04.349358 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b511a5_508f_4ce7_988a_7abc0d8e70da.slice/crio-8b6df450de1d4f0d399413de4be17d66e6ef57ffe913c32518f8f122f6b18d8b WatchSource:0}: Error finding container 8b6df450de1d4f0d399413de4be17d66e6ef57ffe913c32518f8f122f6b18d8b: Status 404 returned error can't find the container with id 8b6df450de1d4f0d399413de4be17d66e6ef57ffe913c32518f8f122f6b18d8b Sep 16 19:09:04.872873 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:04.872838 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzvlv" event={"ID":"10b511a5-508f-4ce7-988a-7abc0d8e70da","Type":"ContainerStarted","Data":"8b6df450de1d4f0d399413de4be17d66e6ef57ffe913c32518f8f122f6b18d8b"} Sep 16 19:09:05.877954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:05.877930 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzvlv" event={"ID":"10b511a5-508f-4ce7-988a-7abc0d8e70da","Type":"ContainerStarted","Data":"ff03aa4325bc3b509447376f41864eb2b2e47a3d06759c367f3b2d5591b92c3c"} Sep 16 19:09:05.908860 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:05.908822 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:06.882947 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:06.882911 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzvlv" event={"ID":"10b511a5-508f-4ce7-988a-7abc0d8e70da","Type":"ContainerStarted","Data":"8093b2a797217b6b297a4f42cc423abca5bdcb45d15cce9d85a2ae1fdaeb649e"} Sep 16 19:09:07.144523 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:07.144437 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:07.348431 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:07.348384 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:07.561335 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:07.561247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:07.665262 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:07.665233 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:27.320871 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:27.320758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:33.863833 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:33.863786 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:51.867695 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:51.867664 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:09:51.874198 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:51.874172 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:09:51.874819 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:09:51.874801 2562 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 19:10:02.711517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:02.711467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:02.800432 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:02.800402 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:14.622754 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:14.622710 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:14.701029 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:14.700994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:17.833494 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:17.833461 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:17.999997 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:17.999951 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:18.210505 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:18.210469 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:36.988485 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:36.988451 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:42.289363 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:42.289321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:53.009506 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:53.009431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:53.139153 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:10:53.139121 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:04.871719 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:04.871678 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:05.236375 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:05.236273 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:30.924420 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:30.924365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:33.527713 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:33.527670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:37.563962 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:37.563923 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:38.130462 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:38.130425 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:38.344634 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:38.344600 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:38.791297 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:38.791264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:38.975930 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:38.975894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:39.114238 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:39.114205 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:39.291694 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:39.291657 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:40.031173 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:40.031138 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:40.429236 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:40.429186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:40.814456 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:40.814374 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:40.973097 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:11:40.973065 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:07.781838 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:07.781804 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:08.054380 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:08.054292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:33.758379 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:33.758322 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:33.935658 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:33.935624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:40.100436 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:40.100402 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:44.365861 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:44.365813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:44.502339 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:44.502303 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:45.704716 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:45.704685 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:45.839254 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:45.839216 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:50.303085 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:50.303049 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:50.483471 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:50.483434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:52.274920 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:52.274887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:52.487565 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:52.487529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:54.457261 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.457206 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mzvlv" podStartSLOduration=481.092011492 podStartE2EDuration="8m2.457189339s" totalImagesPullingTime="1.365177847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:06.908353288 +0000 UTC m=+255.457410562" watchObservedRunningTime="2026-09-16 19:12:54.457189339 +0000 UTC m=+483.006246613" Sep 16 19:12:54.457703 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.457619 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n588x"] Sep 16 19:12:54.457900 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.457883 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4162c94c-63b2-42a9-9906-869b44645018" containerName="registry" Sep 16 19:12:54.457975 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.457902 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4162c94c-63b2-42a9-9906-869b44645018" containerName="registry" Sep 16 19:12:54.458055 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.457978 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4162c94c-63b2-42a9-9906-869b44645018" containerName="registry" Sep 16 19:12:54.460773 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.460753 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:12:54.466250 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.466231 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 19:12:54.475109 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.475090 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-k5dzc\"" Sep 16 19:12:54.484949 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.484928 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 19:12:54.522765 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.522735 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9479229-aceb-42d5-89ef-030cb08314ff-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n588x\" (UID: \"d9479229-aceb-42d5-89ef-030cb08314ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:12:54.522884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.522769 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls5gq\" (UniqueName: \"kubernetes.io/projected/d9479229-aceb-42d5-89ef-030cb08314ff-kube-api-access-ls5gq\") pod \"cert-manager-webhook-6486ffd997-n588x\" (UID: \"d9479229-aceb-42d5-89ef-030cb08314ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:12:54.607188 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.607157 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n588x"] Sep 16 19:12:54.623359 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.623331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9479229-aceb-42d5-89ef-030cb08314ff-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n588x\" (UID: \"d9479229-aceb-42d5-89ef-030cb08314ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:12:54.623496 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.623365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls5gq\" (UniqueName: \"kubernetes.io/projected/d9479229-aceb-42d5-89ef-030cb08314ff-kube-api-access-ls5gq\") pod \"cert-manager-webhook-6486ffd997-n588x\" (UID: \"d9479229-aceb-42d5-89ef-030cb08314ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:12:54.643407 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.643379 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls5gq\" (UniqueName: \"kubernetes.io/projected/d9479229-aceb-42d5-89ef-030cb08314ff-kube-api-access-ls5gq\") pod \"cert-manager-webhook-6486ffd997-n588x\" (UID: \"d9479229-aceb-42d5-89ef-030cb08314ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:12:54.643522 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.643383 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9479229-aceb-42d5-89ef-030cb08314ff-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n588x\" (UID: \"d9479229-aceb-42d5-89ef-030cb08314ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:12:54.657353 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.657324 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:54.769184 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.769098 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:12:54.951038 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:54.950981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:55.083685 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:12:55.083651 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9479229_aceb_42d5_89ef_030cb08314ff.slice/crio-9a379c5c143d7086720eeb5bb535f21d864cc885c2ca189f6e3bb1722b684a32 WatchSource:0}: Error finding container 9a379c5c143d7086720eeb5bb535f21d864cc885c2ca189f6e3bb1722b684a32: Status 404 returned error can't find the container with id 9a379c5c143d7086720eeb5bb535f21d864cc885c2ca189f6e3bb1722b684a32 Sep 16 19:12:55.084509 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:55.084487 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n588x"] Sep 16 19:12:55.085578 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:55.085564 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:12:55.324822 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:55.324787 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:55.609201 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:55.609162 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:55.639407 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:55.639371 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" event={"ID":"d9479229-aceb-42d5-89ef-030cb08314ff","Type":"ContainerStarted","Data":"9a379c5c143d7086720eeb5bb535f21d864cc885c2ca189f6e3bb1722b684a32"} Sep 16 19:12:58.652967 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:58.652882 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" event={"ID":"d9479229-aceb-42d5-89ef-030cb08314ff","Type":"ContainerStarted","Data":"8eecd967fdc7c8d165476385fb6913abf6e68e19c4cd80d46c6c67ea066413e8"} Sep 16 19:12:58.653360 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:58.652993 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:12:58.759736 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:58.759685 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" podStartSLOduration=1.498827199 podStartE2EDuration="4.759669251s" totalImagesPullingTime="3.260842052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:58.757986436 +0000 UTC m=+487.307043709" watchObservedRunningTime="2026-09-16 19:12:58.759669251 +0000 UTC m=+487.308726525" Sep 16 19:12:59.546028 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.545978 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k"] Sep 16 19:12:59.549140 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.549116 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" Sep 16 19:12:59.552439 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.552414 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xkcjx\"" Sep 16 19:12:59.559046 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.558999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgz6d\" (UniqueName: \"kubernetes.io/projected/20a2764b-b46c-47f9-bb4d-c0d164afc8c9-kube-api-access-hgz6d\") pod \"cert-manager-cainjector-559cdd8c87-bwr7k\" (UID: \"20a2764b-b46c-47f9-bb4d-c0d164afc8c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" Sep 16 19:12:59.559179 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.559078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20a2764b-b46c-47f9-bb4d-c0d164afc8c9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bwr7k\" (UID: \"20a2764b-b46c-47f9-bb4d-c0d164afc8c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" Sep 16 19:12:59.574911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.574867 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k"] Sep 16 19:12:59.660034 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.659982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20a2764b-b46c-47f9-bb4d-c0d164afc8c9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bwr7k\" (UID: \"20a2764b-b46c-47f9-bb4d-c0d164afc8c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" Sep 16 19:12:59.660499 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.660081 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgz6d\" (UniqueName: \"kubernetes.io/projected/20a2764b-b46c-47f9-bb4d-c0d164afc8c9-kube-api-access-hgz6d\") pod \"cert-manager-cainjector-559cdd8c87-bwr7k\" (UID: \"20a2764b-b46c-47f9-bb4d-c0d164afc8c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" Sep 16 19:12:59.677583 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.677555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgz6d\" (UniqueName: \"kubernetes.io/projected/20a2764b-b46c-47f9-bb4d-c0d164afc8c9-kube-api-access-hgz6d\") pod \"cert-manager-cainjector-559cdd8c87-bwr7k\" (UID: \"20a2764b-b46c-47f9-bb4d-c0d164afc8c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" Sep 16 19:12:59.698495 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.698461 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20a2764b-b46c-47f9-bb4d-c0d164afc8c9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bwr7k\" (UID: \"20a2764b-b46c-47f9-bb4d-c0d164afc8c9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" Sep 16 19:12:59.858884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:12:59.858855 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" Sep 16 19:13:00.010523 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:00.010492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:00.156845 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:00.156753 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k"] Sep 16 19:13:00.173450 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:13:00.173422 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20a2764b_b46c_47f9_bb4d_c0d164afc8c9.slice/crio-6573c4c7a9109fb2b58795e792599115a0b2da53e172bd88d355f9164645b41c WatchSource:0}: Error finding container 6573c4c7a9109fb2b58795e792599115a0b2da53e172bd88d355f9164645b41c: Status 404 returned error can't find the container with id 6573c4c7a9109fb2b58795e792599115a0b2da53e172bd88d355f9164645b41c Sep 16 19:13:00.446685 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:00.446597 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:00.662327 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:00.662292 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" event={"ID":"20a2764b-b46c-47f9-bb4d-c0d164afc8c9","Type":"ContainerStarted","Data":"958adfff7215686275e878ce46c1a91f83b483c748e4292f738a110ba7dbb66f"} Sep 16 19:13:00.662327 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:00.662333 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" event={"ID":"20a2764b-b46c-47f9-bb4d-c0d164afc8c9","Type":"ContainerStarted","Data":"6573c4c7a9109fb2b58795e792599115a0b2da53e172bd88d355f9164645b41c"} Sep 16 19:13:00.714787 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:00.714688 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bwr7k" podStartSLOduration=1.714672924 podStartE2EDuration="1.714672924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:00.713907589 +0000 UTC m=+489.262964866" watchObservedRunningTime="2026-09-16 19:13:00.714672924 +0000 UTC m=+489.263730198" Sep 16 19:13:01.675680 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:01.675647 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:02.134999 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:02.134963 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:02.672367 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:02.672335 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:04.659897 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:04.659864 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-n588x" Sep 16 19:13:19.803093 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:19.803048 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:20.000380 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:20.000335 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:29.384895 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:29.384863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:33.744083 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:33.744046 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:33.895414 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:33.895381 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb"] Sep 16 19:13:33.899565 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:33.899548 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:33.911121 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:33.911105 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 19:13:33.911838 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:33.911821 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 19:13:33.912912 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:33.912899 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 19:13:33.914532 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:33.914517 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 19:13:33.914799 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:33.914776 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 19:13:33.918857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:33.918844 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wqgrr\"" Sep 16 19:13:34.006037 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.005924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aae3cfa-bcd8-4102-95f6-855769c0d67e-cert\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.006037 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.005970 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fln9w\" (UniqueName: \"kubernetes.io/projected/1aae3cfa-bcd8-4102-95f6-855769c0d67e-kube-api-access-fln9w\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.006281 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.006121 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1aae3cfa-bcd8-4102-95f6-855769c0d67e-metrics-cert\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.006281 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.006153 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1aae3cfa-bcd8-4102-95f6-855769c0d67e-manager-config\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.020430 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.020393 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb"] Sep 16 19:13:34.068862 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.068820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:34.107065 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.107000 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1aae3cfa-bcd8-4102-95f6-855769c0d67e-metrics-cert\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.107224 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.107085 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1aae3cfa-bcd8-4102-95f6-855769c0d67e-manager-config\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.107224 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.107136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aae3cfa-bcd8-4102-95f6-855769c0d67e-cert\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.107224 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.107174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fln9w\" (UniqueName: \"kubernetes.io/projected/1aae3cfa-bcd8-4102-95f6-855769c0d67e-kube-api-access-fln9w\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.107894 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.107871 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1aae3cfa-bcd8-4102-95f6-855769c0d67e-manager-config\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.109601 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.109576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1aae3cfa-bcd8-4102-95f6-855769c0d67e-metrics-cert\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.109696 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.109667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aae3cfa-bcd8-4102-95f6-855769c0d67e-cert\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.126246 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.126219 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fln9w\" (UniqueName: \"kubernetes.io/projected/1aae3cfa-bcd8-4102-95f6-855769c0d67e-kube-api-access-fln9w\") pod \"lws-controller-manager-9498697bd-7v7vb\" (UID: \"1aae3cfa-bcd8-4102-95f6-855769c0d67e\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.208638 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.208601 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:34.550626 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.550538 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb"] Sep 16 19:13:34.569232 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:13:34.569198 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aae3cfa_bcd8_4102_95f6_855769c0d67e.slice/crio-c8d3b38e333467ed90733de6722081e053c0bcfc1dd82c9ecf4117874008d2a1 WatchSource:0}: Error finding container c8d3b38e333467ed90733de6722081e053c0bcfc1dd82c9ecf4117874008d2a1: Status 404 returned error can't find the container with id c8d3b38e333467ed90733de6722081e053c0bcfc1dd82c9ecf4117874008d2a1 Sep 16 19:13:34.790497 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:34.790459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" event={"ID":"1aae3cfa-bcd8-4102-95f6-855769c0d67e","Type":"ContainerStarted","Data":"c8d3b38e333467ed90733de6722081e053c0bcfc1dd82c9ecf4117874008d2a1"} Sep 16 19:13:36.533654 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:36.533613 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:36.715458 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:36.715412 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:37.803153 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:37.803110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" event={"ID":"1aae3cfa-bcd8-4102-95f6-855769c0d67e","Type":"ContainerStarted","Data":"14aef01112fb09aadd57d99d323b5c105f194bcaf3013ebd46060e94c5ce8334"} Sep 16 19:13:37.803575 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:37.803258 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:37.827341 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:37.827288 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" podStartSLOduration=2.193436255 podStartE2EDuration="4.827271693s" totalImagesPullingTime="2.633835438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:37.824861966 +0000 UTC m=+526.373919240" watchObservedRunningTime="2026-09-16 19:13:37.827271693 +0000 UTC m=+526.376328966" Sep 16 19:13:38.554288 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:38.554253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:41.007878 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:41.007844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:41.166700 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:41.166660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:45.160630 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:45.160591 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:45.313745 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:45.313657 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:48.251447 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:48.251413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:48.809357 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:48.809330 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9498697bd-7v7vb" Sep 16 19:13:52.575819 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:52.575772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:52.933134 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:52.933096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:53.050509 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:13:53.050470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:09.560299 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:09.560261 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:12.405004 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:12.404970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:12.830602 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:12.830504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:13.295391 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:13.295350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:15.139417 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:15.139378 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:16.131387 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:16.131353 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:16.926784 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:16.926712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:17.371249 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:17.371158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:18.063335 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:18.063300 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:18.899854 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:18.899816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:20.029492 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:20.029451 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:21.175451 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.175403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:21.547252 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.547174 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs"] Sep 16 19:14:21.550214 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.550199 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" Sep 16 19:14:21.554899 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.554878 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 19:14:21.555041 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.554925 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 16 19:14:21.555906 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.555887 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-hc8sj\"" Sep 16 19:14:21.556001 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.555933 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 19:14:21.578588 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.578564 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs"] Sep 16 19:14:21.658694 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.658657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzwn\" (UniqueName: \"kubernetes.io/projected/66a682f5-23a0-4f9d-88b5-1937b74f3b3d-kube-api-access-wmzwn\") pod \"dns-operator-controller-manager-65dc8f76f5-7l5bs\" (UID: \"66a682f5-23a0-4f9d-88b5-1937b74f3b3d\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" Sep 16 19:14:21.760002 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.759966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzwn\" (UniqueName: \"kubernetes.io/projected/66a682f5-23a0-4f9d-88b5-1937b74f3b3d-kube-api-access-wmzwn\") pod \"dns-operator-controller-manager-65dc8f76f5-7l5bs\" (UID: \"66a682f5-23a0-4f9d-88b5-1937b74f3b3d\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" Sep 16 19:14:21.776658 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.776626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzwn\" (UniqueName: \"kubernetes.io/projected/66a682f5-23a0-4f9d-88b5-1937b74f3b3d-kube-api-access-wmzwn\") pod \"dns-operator-controller-manager-65dc8f76f5-7l5bs\" (UID: \"66a682f5-23a0-4f9d-88b5-1937b74f3b3d\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" Sep 16 19:14:21.859586 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:21.859547 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" Sep 16 19:14:22.137699 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:22.137676 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs"] Sep 16 19:14:22.140033 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:14:22.139984 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a682f5_23a0_4f9d_88b5_1937b74f3b3d.slice/crio-bdc0277a617eb4e9f8d4149cfb8ce6f97e2c7edffc55082218b09a3bfe64ba0a WatchSource:0}: Error finding container bdc0277a617eb4e9f8d4149cfb8ce6f97e2c7edffc55082218b09a3bfe64ba0a: Status 404 returned error can't find the container with id bdc0277a617eb4e9f8d4149cfb8ce6f97e2c7edffc55082218b09a3bfe64ba0a Sep 16 19:14:22.964488 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:22.964457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" event={"ID":"66a682f5-23a0-4f9d-88b5-1937b74f3b3d","Type":"ContainerStarted","Data":"bdc0277a617eb4e9f8d4149cfb8ce6f97e2c7edffc55082218b09a3bfe64ba0a"} Sep 16 19:14:23.181311 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:23.181271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:23.402251 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:23.402218 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:24.274075 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.274038 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv"] Sep 16 19:14:24.275683 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.275636 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:24.285395 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.285373 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.289285 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.289259 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7wrrl\"" Sep 16 19:14:24.289446 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.289424 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 16 19:14:24.290228 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.290208 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 16 19:14:24.293065 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.293041 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv"] Sep 16 19:14:24.378719 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.378680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f1b44d4-bd18-4c46-a596-9910d87ce890-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-qlvpv\" (UID: \"3f1b44d4-bd18-4c46-a596-9910d87ce890\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.378904 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.378754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f1b44d4-bd18-4c46-a596-9910d87ce890-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qlvpv\" (UID: \"3f1b44d4-bd18-4c46-a596-9910d87ce890\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.378904 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.378802 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55q6n\" (UniqueName: \"kubernetes.io/projected/3f1b44d4-bd18-4c46-a596-9910d87ce890-kube-api-access-55q6n\") pod \"kuadrant-console-plugin-fbdd4f445-qlvpv\" (UID: \"3f1b44d4-bd18-4c46-a596-9910d87ce890\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.480061 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.480007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f1b44d4-bd18-4c46-a596-9910d87ce890-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qlvpv\" (UID: \"3f1b44d4-bd18-4c46-a596-9910d87ce890\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.480247 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.480099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55q6n\" (UniqueName: \"kubernetes.io/projected/3f1b44d4-bd18-4c46-a596-9910d87ce890-kube-api-access-55q6n\") pod \"kuadrant-console-plugin-fbdd4f445-qlvpv\" (UID: \"3f1b44d4-bd18-4c46-a596-9910d87ce890\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.480313 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.480237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f1b44d4-bd18-4c46-a596-9910d87ce890-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-qlvpv\" (UID: \"3f1b44d4-bd18-4c46-a596-9910d87ce890\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.480908 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.480886 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f1b44d4-bd18-4c46-a596-9910d87ce890-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-qlvpv\" (UID: \"3f1b44d4-bd18-4c46-a596-9910d87ce890\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.482766 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.482740 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f1b44d4-bd18-4c46-a596-9910d87ce890-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qlvpv\" (UID: \"3f1b44d4-bd18-4c46-a596-9910d87ce890\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.496979 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.496958 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55q6n\" (UniqueName: \"kubernetes.io/projected/3f1b44d4-bd18-4c46-a596-9910d87ce890-kube-api-access-55q6n\") pod \"kuadrant-console-plugin-fbdd4f445-qlvpv\" (UID: \"3f1b44d4-bd18-4c46-a596-9910d87ce890\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.500345 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.500272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:24.597149 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.597115 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" Sep 16 19:14:24.962721 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.962544 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv"] Sep 16 19:14:24.965128 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:14:24.965092 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f1b44d4_bd18_4c46_a596_9910d87ce890.slice/crio-c36e22ef3b847ca4ba3c7ff8006be13d80f6649849f7e9a696b4e2a163199180 WatchSource:0}: Error finding container c36e22ef3b847ca4ba3c7ff8006be13d80f6649849f7e9a696b4e2a163199180: Status 404 returned error can't find the container with id c36e22ef3b847ca4ba3c7ff8006be13d80f6649849f7e9a696b4e2a163199180 Sep 16 19:14:24.980296 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.980269 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" event={"ID":"3f1b44d4-bd18-4c46-a596-9910d87ce890","Type":"ContainerStarted","Data":"c36e22ef3b847ca4ba3c7ff8006be13d80f6649849f7e9a696b4e2a163199180"} Sep 16 19:14:24.981857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.981828 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" event={"ID":"66a682f5-23a0-4f9d-88b5-1937b74f3b3d","Type":"ContainerStarted","Data":"89980cd9a0515e4a7da7ec5f574b2ced41630a1b511915acf1a18fbadf055a6a"} Sep 16 19:14:24.981978 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:24.981962 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" Sep 16 19:14:25.013239 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:25.013200 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" podStartSLOduration=1.420276924 podStartE2EDuration="4.013184188s" totalImagesPullingTime="2.592907264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:25.012797107 +0000 UTC m=+573.561854382" watchObservedRunningTime="2026-09-16 19:14:25.013184188 +0000 UTC m=+573.562241461" Sep 16 19:14:26.157960 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:26.157929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:26.291478 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:26.291447 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:27.510159 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:27.510130 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:27.703200 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:27.703161 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:27.874461 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:27.874432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:35.989263 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:35.989234 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-7l5bs" Sep 16 19:14:36.273507 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:36.273406 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:36.503713 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:36.503656 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:38.542385 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:38.542347 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:38.660538 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:38.660499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:40.037720 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:40.037685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" event={"ID":"3f1b44d4-bd18-4c46-a596-9910d87ce890","Type":"ContainerStarted","Data":"5966e6fb5b97b76e5ac947fcbe7da86f4cd7ff8b0fd3fd472f8beb89a1e106ee"} Sep 16 19:14:40.064244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:40.064193 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qlvpv" podStartSLOduration=1.71641387 podStartE2EDuration="16.064177156s" totalImagesPullingTime="14.347763286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:40.063581292 +0000 UTC m=+588.612638567" watchObservedRunningTime="2026-09-16 19:14:40.064177156 +0000 UTC m=+588.613234430" Sep 16 19:14:42.864537 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:42.864499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:43.024622 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:43.024580 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:43.996172 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:43.996135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:44.143941 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:44.143904 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:46.800645 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:46.800609 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:46.930312 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:46.930270 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:47.142821 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:47.142791 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:47.446718 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:47.446625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:51.893132 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:51.893104 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:14:51.894499 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:51.894478 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:14:55.482892 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:55.482845 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:55.602438 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:55.602403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:55.806663 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:55.806568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:57.951694 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:57.951661 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:58.052865 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:14:58.052828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:03.274489 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:03.274447 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:05.563189 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.563138 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:05.579806 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.579770 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hd896"] Sep 16 19:15:05.582805 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.582785 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:05.585478 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.585460 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 19:15:05.595401 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.595381 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hd896"] Sep 16 19:15:05.673111 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.673083 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hd896"] Sep 16 19:15:05.676572 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.676544 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pm4h\" (UniqueName: \"kubernetes.io/projected/648776fc-7de4-4d9c-90d6-3fce1d5db52a-kube-api-access-2pm4h\") pod \"limitador-limitador-8588749977-hd896\" (UID: \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\") " pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:05.676693 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.676583 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/648776fc-7de4-4d9c-90d6-3fce1d5db52a-config-file\") pod \"limitador-limitador-8588749977-hd896\" (UID: \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\") " pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:05.687634 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.687596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:05.777471 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.777431 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/648776fc-7de4-4d9c-90d6-3fce1d5db52a-config-file\") pod \"limitador-limitador-8588749977-hd896\" (UID: \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\") " pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:05.777647 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.777501 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pm4h\" (UniqueName: \"kubernetes.io/projected/648776fc-7de4-4d9c-90d6-3fce1d5db52a-kube-api-access-2pm4h\") pod \"limitador-limitador-8588749977-hd896\" (UID: \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\") " pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:05.778084 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.778066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/648776fc-7de4-4d9c-90d6-3fce1d5db52a-config-file\") pod \"limitador-limitador-8588749977-hd896\" (UID: \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\") " pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:05.794830 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.794806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pm4h\" (UniqueName: \"kubernetes.io/projected/648776fc-7de4-4d9c-90d6-3fce1d5db52a-kube-api-access-2pm4h\") pod \"limitador-limitador-8588749977-hd896\" (UID: \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\") " pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:05.833541 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.833469 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:05.892832 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:05.892801 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:06.164689 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.164527 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hd896"] Sep 16 19:15:06.167236 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:15:06.167206 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648776fc_7de4_4d9c_90d6_3fce1d5db52a.slice/crio-3a086182ec384b98d721ea04ed18cb778ecb6ae7284301eba2048f6fe4c384b5 WatchSource:0}: Error finding container 3a086182ec384b98d721ea04ed18cb778ecb6ae7284301eba2048f6fe4c384b5: Status 404 returned error can't find the container with id 3a086182ec384b98d721ea04ed18cb778ecb6ae7284301eba2048f6fe4c384b5 Sep 16 19:15:06.607542 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.607508 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:06.728982 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.728948 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-xvqk4"] Sep 16 19:15:06.733895 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.733868 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-xvqk4" Sep 16 19:15:06.734523 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.734474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:06.736690 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.736673 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4df5s\"" Sep 16 19:15:06.744131 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.744110 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xvqk4"] Sep 16 19:15:06.786330 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.786296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbc48\" (UniqueName: \"kubernetes.io/projected/7785f1f2-6670-44b8-90a6-3d466c5a4d4a-kube-api-access-vbc48\") pod \"authorino-59c974bd48-xvqk4\" (UID: \"7785f1f2-6670-44b8-90a6-3d466c5a4d4a\") " pod="kuadrant-system/authorino-59c974bd48-xvqk4" Sep 16 19:15:06.887383 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.887298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbc48\" (UniqueName: \"kubernetes.io/projected/7785f1f2-6670-44b8-90a6-3d466c5a4d4a-kube-api-access-vbc48\") pod \"authorino-59c974bd48-xvqk4\" (UID: \"7785f1f2-6670-44b8-90a6-3d466c5a4d4a\") " pod="kuadrant-system/authorino-59c974bd48-xvqk4" Sep 16 19:15:06.897322 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:06.897291 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbc48\" (UniqueName: \"kubernetes.io/projected/7785f1f2-6670-44b8-90a6-3d466c5a4d4a-kube-api-access-vbc48\") pod \"authorino-59c974bd48-xvqk4\" (UID: \"7785f1f2-6670-44b8-90a6-3d466c5a4d4a\") " pod="kuadrant-system/authorino-59c974bd48-xvqk4" Sep 16 19:15:07.042943 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:07.042906 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-xvqk4" Sep 16 19:15:07.141878 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:07.141799 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-hd896" event={"ID":"648776fc-7de4-4d9c-90d6-3fce1d5db52a","Type":"ContainerStarted","Data":"3a086182ec384b98d721ea04ed18cb778ecb6ae7284301eba2048f6fe4c384b5"} Sep 16 19:15:07.316756 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:07.316724 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xvqk4"] Sep 16 19:15:07.473889 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:15:07.473809 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7785f1f2_6670_44b8_90a6_3d466c5a4d4a.slice/crio-4a4ed246c9d7113b5b528736470b1f3a697b17ad8edf92d4328235ed56186b76 WatchSource:0}: Error finding container 4a4ed246c9d7113b5b528736470b1f3a697b17ad8edf92d4328235ed56186b76: Status 404 returned error can't find the container with id 4a4ed246c9d7113b5b528736470b1f3a697b17ad8edf92d4328235ed56186b76 Sep 16 19:15:08.146826 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:08.146791 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-hd896" event={"ID":"648776fc-7de4-4d9c-90d6-3fce1d5db52a","Type":"ContainerStarted","Data":"f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538"} Sep 16 19:15:08.147264 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:08.146897 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:08.147968 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:08.147946 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-xvqk4" event={"ID":"7785f1f2-6670-44b8-90a6-3d466c5a4d4a","Type":"ContainerStarted","Data":"4a4ed246c9d7113b5b528736470b1f3a697b17ad8edf92d4328235ed56186b76"} Sep 16 19:15:08.170539 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:08.170494 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-hd896" podStartSLOduration=1.8130976890000001 podStartE2EDuration="3.170479523s" totalImagesPullingTime="1.357381834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:08.170094393 +0000 UTC m=+616.719151667" watchObservedRunningTime="2026-09-16 19:15:08.170479523 +0000 UTC m=+616.719536855" Sep 16 19:15:11.162071 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:11.162031 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-xvqk4" event={"ID":"7785f1f2-6670-44b8-90a6-3d466c5a4d4a","Type":"ContainerStarted","Data":"2f7b06a41c1681af509b18d3587ec9e717fb45820fecb9cef476024744a2e020"} Sep 16 19:15:11.202247 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:11.202190 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-xvqk4" podStartSLOduration=1.7029145799999998 podStartE2EDuration="5.202170224s" totalImagesPullingTime="3.499255644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:11.200919937 +0000 UTC m=+619.749977213" watchObservedRunningTime="2026-09-16 19:15:11.202170224 +0000 UTC m=+619.751227497" Sep 16 19:15:19.152517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:19.152440 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:20.108207 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:20.108172 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hd896"] Sep 16 19:15:20.108397 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:20.108377 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-hd896" podUID="648776fc-7de4-4d9c-90d6-3fce1d5db52a" containerName="limitador" containerID="cri-o://f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538" gracePeriod=30 Sep 16 19:15:20.652773 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:20.652750 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:20.788901 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:20.788821 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pm4h\" (UniqueName: \"kubernetes.io/projected/648776fc-7de4-4d9c-90d6-3fce1d5db52a-kube-api-access-2pm4h\") pod \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\" (UID: \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\") " Sep 16 19:15:20.788901 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:20.788867 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/648776fc-7de4-4d9c-90d6-3fce1d5db52a-config-file\") pod \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\" (UID: \"648776fc-7de4-4d9c-90d6-3fce1d5db52a\") " Sep 16 19:15:20.789262 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:20.789240 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648776fc-7de4-4d9c-90d6-3fce1d5db52a-config-file" (OuterVolumeSpecName: "config-file") pod "648776fc-7de4-4d9c-90d6-3fce1d5db52a" (UID: "648776fc-7de4-4d9c-90d6-3fce1d5db52a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:15:20.791046 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:20.791007 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648776fc-7de4-4d9c-90d6-3fce1d5db52a-kube-api-access-2pm4h" (OuterVolumeSpecName: "kube-api-access-2pm4h") pod "648776fc-7de4-4d9c-90d6-3fce1d5db52a" (UID: "648776fc-7de4-4d9c-90d6-3fce1d5db52a"). InnerVolumeSpecName "kube-api-access-2pm4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:15:20.890209 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:20.890157 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pm4h\" (UniqueName: \"kubernetes.io/projected/648776fc-7de4-4d9c-90d6-3fce1d5db52a-kube-api-access-2pm4h\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:15:20.890209 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:20.890203 2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/648776fc-7de4-4d9c-90d6-3fce1d5db52a-config-file\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:15:21.203292 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.203258 2562 generic.go:356] "Generic (PLEG): container finished" podID="648776fc-7de4-4d9c-90d6-3fce1d5db52a" containerID="f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538" exitCode=0 Sep 16 19:15:21.203292 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.203295 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-hd896" Sep 16 19:15:21.203490 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.203348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-hd896" event={"ID":"648776fc-7de4-4d9c-90d6-3fce1d5db52a","Type":"ContainerDied","Data":"f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538"} Sep 16 19:15:21.203490 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.203397 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-hd896" event={"ID":"648776fc-7de4-4d9c-90d6-3fce1d5db52a","Type":"ContainerDied","Data":"3a086182ec384b98d721ea04ed18cb778ecb6ae7284301eba2048f6fe4c384b5"} Sep 16 19:15:21.203490 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.203418 2562 scope.go:117] "RemoveContainer" containerID="f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538" Sep 16 19:15:21.211514 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.211493 2562 scope.go:117] "RemoveContainer" containerID="f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538" Sep 16 19:15:21.211766 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:15:21.211745 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538\": container with ID starting with f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538 not found: ID does not exist" containerID="f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538" Sep 16 19:15:21.211808 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.211773 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538"} err="failed to get container status \"f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538\": rpc error: code = NotFound desc = could not find container \"f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538\": container with ID starting with f765860ad3b8dc98719c1abf1468b3a1b8690cb5ca7d7f2b98e800d02f16c538 not found: ID does not exist" Sep 16 19:15:21.233502 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.233474 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hd896"] Sep 16 19:15:21.238358 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.238337 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hd896"] Sep 16 19:15:21.962721 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:21.962690 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="648776fc-7de4-4d9c-90d6-3fce1d5db52a" path="/var/lib/kubelet/pods/648776fc-7de4-4d9c-90d6-3fce1d5db52a/volumes" Sep 16 19:15:26.266228 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:26.266196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:26.597060 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:26.597003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:30.899864 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:30.899827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:32.130417 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:32.130376 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xvqk4"] Sep 16 19:15:32.130786 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:32.130547 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-xvqk4" podUID="7785f1f2-6670-44b8-90a6-3d466c5a4d4a" containerName="authorino" containerID="cri-o://2f7b06a41c1681af509b18d3587ec9e717fb45820fecb9cef476024744a2e020" gracePeriod=30 Sep 16 19:15:32.250990 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:32.250954 2562 generic.go:356] "Generic (PLEG): container finished" podID="7785f1f2-6670-44b8-90a6-3d466c5a4d4a" containerID="2f7b06a41c1681af509b18d3587ec9e717fb45820fecb9cef476024744a2e020" exitCode=0 Sep 16 19:15:32.251139 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:32.251037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-xvqk4" event={"ID":"7785f1f2-6670-44b8-90a6-3d466c5a4d4a","Type":"ContainerDied","Data":"2f7b06a41c1681af509b18d3587ec9e717fb45820fecb9cef476024744a2e020"} Sep 16 19:15:32.362450 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:32.362428 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-xvqk4" Sep 16 19:15:32.479119 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:32.479028 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbc48\" (UniqueName: \"kubernetes.io/projected/7785f1f2-6670-44b8-90a6-3d466c5a4d4a-kube-api-access-vbc48\") pod \"7785f1f2-6670-44b8-90a6-3d466c5a4d4a\" (UID: \"7785f1f2-6670-44b8-90a6-3d466c5a4d4a\") " Sep 16 19:15:32.481034 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:32.480994 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7785f1f2-6670-44b8-90a6-3d466c5a4d4a-kube-api-access-vbc48" (OuterVolumeSpecName: "kube-api-access-vbc48") pod "7785f1f2-6670-44b8-90a6-3d466c5a4d4a" (UID: "7785f1f2-6670-44b8-90a6-3d466c5a4d4a"). InnerVolumeSpecName "kube-api-access-vbc48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:15:32.580190 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:32.580145 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbc48\" (UniqueName: \"kubernetes.io/projected/7785f1f2-6670-44b8-90a6-3d466c5a4d4a-kube-api-access-vbc48\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:15:33.255782 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:33.255756 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-xvqk4" Sep 16 19:15:33.255782 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:33.255760 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-xvqk4" event={"ID":"7785f1f2-6670-44b8-90a6-3d466c5a4d4a","Type":"ContainerDied","Data":"4a4ed246c9d7113b5b528736470b1f3a697b17ad8edf92d4328235ed56186b76"} Sep 16 19:15:33.256265 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:33.255803 2562 scope.go:117] "RemoveContainer" containerID="2f7b06a41c1681af509b18d3587ec9e717fb45820fecb9cef476024744a2e020" Sep 16 19:15:33.279878 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:33.279850 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xvqk4"] Sep 16 19:15:33.281743 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:33.281715 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xvqk4"] Sep 16 19:15:33.961707 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:33.961677 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7785f1f2-6670-44b8-90a6-3d466c5a4d4a" path="/var/lib/kubelet/pods/7785f1f2-6670-44b8-90a6-3d466c5a4d4a/volumes" Sep 16 19:15:35.953911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:35.953873 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:36.059053 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:36.059003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:45.008928 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:45.008890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:54.775457 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.775425 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt"] Sep 16 19:15:54.775857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.775679 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648776fc-7de4-4d9c-90d6-3fce1d5db52a" containerName="limitador" Sep 16 19:15:54.775857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.775689 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="648776fc-7de4-4d9c-90d6-3fce1d5db52a" containerName="limitador" Sep 16 19:15:54.775857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.775706 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7785f1f2-6670-44b8-90a6-3d466c5a4d4a" containerName="authorino" Sep 16 19:15:54.775857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.775711 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7785f1f2-6670-44b8-90a6-3d466c5a4d4a" containerName="authorino" Sep 16 19:15:54.775857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.775749 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="648776fc-7de4-4d9c-90d6-3fce1d5db52a" containerName="limitador" Sep 16 19:15:54.775857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.775757 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7785f1f2-6670-44b8-90a6-3d466c5a4d4a" containerName="authorino" Sep 16 19:15:54.779811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.779793 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:54.795384 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.795359 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 16 19:15:54.795554 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.795535 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 16 19:15:54.795716 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.795695 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-brh4c\"" Sep 16 19:15:54.795831 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.795813 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 19:15:54.795948 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.795933 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 19:15:54.796032 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.795933 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 16 19:15:54.798618 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.798602 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 19:15:54.844558 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.844526 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt"] Sep 16 19:15:54.951037 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.950983 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:54.951210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.951055 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:54.951210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.951096 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:54.951210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.951164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e84ed16-bdae-4d25-a906-dc6785378da2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:54.951210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.951208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:54.951359 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.951229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e84ed16-bdae-4d25-a906-dc6785378da2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:54.951359 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:54.951247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcjr\" (UniqueName: \"kubernetes.io/projected/6e84ed16-bdae-4d25-a906-dc6785378da2-kube-api-access-vfcjr\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.051980 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.051900 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e84ed16-bdae-4d25-a906-dc6785378da2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.051980 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.051950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.051980 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.051980 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e84ed16-bdae-4d25-a906-dc6785378da2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.052257 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.052004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcjr\" (UniqueName: \"kubernetes.io/projected/6e84ed16-bdae-4d25-a906-dc6785378da2-kube-api-access-vfcjr\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.052257 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.052072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.052257 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.052100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.052257 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.052141 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.052842 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.052819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.054390 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.054368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e84ed16-bdae-4d25-a906-dc6785378da2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.054468 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.054448 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e84ed16-bdae-4d25-a906-dc6785378da2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.054831 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.054808 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.054872 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.054814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.071077 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.071050 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e84ed16-bdae-4d25-a906-dc6785378da2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.080021 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.079989 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcjr\" (UniqueName: \"kubernetes.io/projected/6e84ed16-bdae-4d25-a906-dc6785378da2-kube-api-access-vfcjr\") pod \"istiod-openshift-gateway-5677ccb49b-t8llt\" (UID: \"6e84ed16-bdae-4d25-a906-dc6785378da2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.088806 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.088788 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:55.429879 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:55.426303 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt"] Sep 16 19:15:55.433628 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:15:55.433601 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e84ed16_bdae_4d25_a906_dc6785378da2.slice/crio-c3a96932357f97f6378687c456e4958b538bf0905e87864be6cecd1cf0f855b0 WatchSource:0}: Error finding container c3a96932357f97f6378687c456e4958b538bf0905e87864be6cecd1cf0f855b0: Status 404 returned error can't find the container with id c3a96932357f97f6378687c456e4958b538bf0905e87864be6cecd1cf0f855b0 Sep 16 19:15:56.345315 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:56.345258 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" event={"ID":"6e84ed16-bdae-4d25-a906-dc6785378da2","Type":"ContainerStarted","Data":"c3a96932357f97f6378687c456e4958b538bf0905e87864be6cecd1cf0f855b0"} Sep 16 19:15:57.767950 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:57.767910 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 19:15:57.768197 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:57.767983 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 19:15:58.357810 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:58.357685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" event={"ID":"6e84ed16-bdae-4d25-a906-dc6785378da2","Type":"ContainerStarted","Data":"b796785bce5420e25076954a6a56d3305f6593d0f916238287684bd6252dfbad"} Sep 16 19:15:58.358421 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:58.358180 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:15:58.359717 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:58.359679 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-t8llt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 19:15:58.359823 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:58.359748 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" podUID="6e84ed16-bdae-4d25-a906-dc6785378da2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:15:58.397522 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:58.397438 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" podStartSLOduration=2.065138176 podStartE2EDuration="4.397414538s" totalImagesPullingTime="2.332276362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:58.396912362 +0000 UTC m=+666.945969635" watchObservedRunningTime="2026-09-16 19:15:58.397414538 +0000 UTC m=+666.946471814" Sep 16 19:15:59.362188 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:15:59.362158 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-t8llt" Sep 16 19:16:01.175861 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:01.175827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:01.289994 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:01.289959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:03.088223 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:03.088160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:03.197455 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:03.197409 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:14.310891 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:14.310855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:14.439250 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:14.439216 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:14.599884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:14.599835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:14.806983 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:14.806949 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:15.415318 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:15.415271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:15.554271 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:15.554234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:15.680211 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:15.680117 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:53.440728 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:16:53.440645 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:07.156571 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:17:07.156528 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:17.134171 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:17:17.134136 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:17.277746 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:17:17.277685 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:37.264476 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:17:37.264431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:37.398358 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:17:37.398312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:37.565370 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:17:37.565271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:37.665902 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:17:37.665858 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:39.197290 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:17:39.197250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:39.296484 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:17:39.296439 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:14.849393 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:14.849355 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:14.956776 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:14.956740 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:28.003904 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:28.003869 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:28.210553 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:28.210523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:46.437272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:46.437190 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:46.780549 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:46.780463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:46.969202 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:46.969168 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:47.524828 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:47.524792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:51.915816 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:51.915789 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:19:51.916253 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:51.916141 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:19:57.137263 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:19:57.137226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:20:17.968468 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:20:17.968439 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:20:18.242262 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:20:18.242186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:20:30.917650 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:20:30.917608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:21:09.249297 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:21:09.249257 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:21:09.410411 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:21:09.410377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:21:40.445416 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:21:40.445381 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:21:47.701541 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:21:47.701509 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:21:59.151882 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:21:59.151849 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:00.728809 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:00.728725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:04.149963 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:04.149932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:04.533463 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:04.533362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:06.478818 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:06.478780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:07.524833 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:07.524804 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:07.956063 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:07.956028 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:08.450527 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:08.450494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:10.746149 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:10.746113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:12.364744 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:12.364699 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:18.358954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:18.358893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:18.503881 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:18.503844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:43.763824 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:43.763786 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:43.871412 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:43.871375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:56.896032 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:56.895936 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:57.147112 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:57.146999 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:59.092224 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:59.092186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:59.274550 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:59.274517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:59.425407 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:59.425324 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:22:59.862678 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:22:59.862647 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:00.164088 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:00.163994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:05.019746 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:05.019709 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:10.310613 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:10.310578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:10.458326 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:10.458290 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:10.876305 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:10.876266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:11.002213 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:11.002179 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:12.166124 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:12.166089 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:12.273728 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:12.273695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:13.515859 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:13.515828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:13.863486 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:13.863441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:26.272090 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:26.272055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:33.906368 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:33.906330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:34.174982 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:34.174906 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:34.567291 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:34.567256 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:34.783169 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:34.783080 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:39.380463 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:39.380425 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:42.045610 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:42.045576 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:42.865503 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:42.865472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:42.987023 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:42.986989 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:43.150227 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:43.150143 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:43.474312 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:43.474237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:43.595564 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:43.595535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:45.224457 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:45.224419 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:46.618982 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:46.618944 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:46.796868 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:46.796840 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:48.029406 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:48.029376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:52.509414 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:52.509335 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:52.702104 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:52.702069 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:53.025292 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:53.025264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:55.292078 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:55.292042 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:55.417049 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:55.417003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:57.106342 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:57.106286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:57.418252 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:57.418167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:57.812358 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:23:57.812279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:00.371382 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:00.371347 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:03.901678 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:03.901644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:04.680603 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:04.680570 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:04.809485 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:04.809455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:06.597430 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:06.597398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:07.269594 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:07.269559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:07.599052 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:07.599007 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:08.344410 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:08.344373 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:10.253684 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:10.253646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:10.382907 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:10.382873 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:13.022953 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:13.022920 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:14.490569 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:14.490483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:14.607880 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:14.607848 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:17.845437 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:17.845400 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:46.114984 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:46.114955 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:51.937758 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:51.937722 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:24:51.938770 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:51.938750 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:24:58.296775 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:58.296744 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:58.410210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:58.410178 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:58.514176 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:24:58.514142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:02.862596 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:02.862559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:02.954448 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:02.954416 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:03.225943 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:03.225856 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:03.512433 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:03.512345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:06.335971 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:06.335942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:13.003953 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:13.003918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:13.117122 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:13.117087 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:15.473308 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:15.473277 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:26.123680 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:26.123649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:37.719415 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:37.719379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:25:37.838243 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:25:37.838214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:12.206751 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:26:12.206717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:12.311695 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:26:12.311661 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:27.665574 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:26:27.665534 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:27.802830 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:26:27.802795 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:33.749648 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:26:33.749614 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:34.120697 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:26:34.120660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:35.663592 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:26:35.663560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:42.191323 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:26:42.191288 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:42.279181 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:26:42.279144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:02.168726 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:02.168690 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:03.425129 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:03.425091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:05.409136 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:05.409103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:09.808065 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:09.808033 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:16.261953 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:16.261871 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:16.364132 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:16.364102 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:20.627562 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:20.627527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:20.886352 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:20.886256 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:23.373758 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:23.373725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:23.465950 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:23.465918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:27:31.569563 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:27:31.569529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:13.459899 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:13.459864 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:13.563545 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:13.563511 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:20.451104 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:20.451069 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:23.550587 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:23.550556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:23.865704 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:23.865668 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:27.256630 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:27.256597 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:27.469150 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:27.469116 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:34.894210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:34.894171 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:35.187389 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:35.187308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:57.719922 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:57.719845 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:28:57.807156 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:28:57.807121 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:29:12.423653 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:29:12.423623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:29:12.522090 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:29:12.522051 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:29:47.789183 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:29:47.789137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:29:47.948941 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:29:47.948909 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:29:50.229085 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:29:50.229049 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:29:50.871304 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:29:50.871254 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:29:51.958759 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:29:51.958732 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:29:51.961382 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:29:51.961360 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:29:55.278087 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:29:55.278055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:01.465933 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:01.465904 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:01.612177 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:01.612144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:01.737812 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:01.737734 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:02.274355 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:02.274311 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:02.372363 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:02.372301 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:08.770646 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:08.770613 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:12.393709 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:12.393669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:15.920328 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:15.920245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:16.108432 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:16.108397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:24.948827 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:24.948794 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:25.225301 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:25.225215 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:36.442940 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:36.442903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:38.031663 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:38.031626 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:38.123916 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:38.123882 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:30:55.101365 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:30:55.101329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:31:01.541693 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:01.541654 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:31:24.421777 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:24.421743 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:31:24.528299 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:24.528261 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:31:31.763871 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:31.763829 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:31:36.427914 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.427881 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk"] Sep 16 19:31:36.431067 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.431051 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.443165 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.443143 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\"" Sep 16 19:31:36.443274 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.443169 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\"" Sep 16 19:31:36.448100 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.448084 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-mr2jf\"" Sep 16 19:31:36.448625 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.448610 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-tr6cv\"" Sep 16 19:31:36.449522 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.449509 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\"" Sep 16 19:31:36.498944 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.498917 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk"] Sep 16 19:31:36.506660 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.506635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.506794 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.506665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-home\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.506794 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.506683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.506794 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.506708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.506794 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.506779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.506927 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.506834 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lpv\" (UniqueName: \"kubernetes.io/projected/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-kube-api-access-n4lpv\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.534325 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.534297 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:31:36.607362 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.607326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.607362 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.607361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-home\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.607609 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.607379 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.607609 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.607492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.607609 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.607535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.607609 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.607591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lpv\" (UniqueName: \"kubernetes.io/projected/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-kube-api-access-n4lpv\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.607820 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.607741 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-home\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.607820 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.607787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.607891 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.607863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.609843 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.609812 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.609972 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.609937 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.620802 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.620777 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lpv\" (UniqueName: \"kubernetes.io/projected/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-kube-api-access-n4lpv\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.740161 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.740070 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:36.894910 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:36.894875 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:31:37.036870 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:37.036846 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk"] Sep 16 19:31:37.038719 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:31:37.038694 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cc27c1_a832_49a7_9dec_3dff51cba1cb.slice/crio-b3877fac7cf44d9a79dd0560c92cd7eac566537a6999182f3816f41c44612524 WatchSource:0}: Error finding container b3877fac7cf44d9a79dd0560c92cd7eac566537a6999182f3816f41c44612524: Status 404 returned error can't find the container with id b3877fac7cf44d9a79dd0560c92cd7eac566537a6999182f3816f41c44612524 Sep 16 19:31:37.040793 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:37.040779 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:31:37.850994 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:37.850954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" event={"ID":"d3cc27c1-a832-49a7-9dec-3dff51cba1cb","Type":"ContainerStarted","Data":"b3877fac7cf44d9a79dd0560c92cd7eac566537a6999182f3816f41c44612524"} Sep 16 19:31:38.855964 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:38.855922 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" event={"ID":"d3cc27c1-a832-49a7-9dec-3dff51cba1cb","Type":"ContainerStarted","Data":"9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265"} Sep 16 19:31:38.856360 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:38.856053 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:40.757630 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:40.757539 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:31:40.865124 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:40.865088 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" event={"ID":"d3cc27c1-a832-49a7-9dec-3dff51cba1cb","Type":"ContainerStarted","Data":"f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518"} Sep 16 19:31:40.920564 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:40.920511 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" podStartSLOduration=2.000327491 podStartE2EDuration="4.920496755s" totalImagesPullingTime="2.920169264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:40.920394448 +0000 UTC m=+1609.469451721" watchObservedRunningTime="2026-09-16 19:31:40.920496755 +0000 UTC m=+1609.469554334" Sep 16 19:31:40.925846 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:40.925819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:31:41.740464 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:41.740428 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:46.740981 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:46.740904 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:46.752878 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:46.752848 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:46.897601 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:46.897570 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:31:57.902765 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:31:57.902732 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:32:11.747499 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:32:11.747463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:32:30.165331 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:32:30.165295 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:32:30.316910 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:32:30.316870 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:20.164464 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:20.164375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:20.268575 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:20.268541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:20.363589 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:20.363553 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:27.146887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:27.146849 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:27.243360 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:27.243311 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:33.630793 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:33.630757 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:33.736122 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:33.736091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:54.048095 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:54.048056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:54.353402 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:54.353366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:56.389509 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:56.389475 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:56.418518 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:56.418440 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk"] Sep 16 19:33:56.434783 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:33:56.434709 2562 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:33:56.434783 ip-10-0-139-17 kubenswrapper[2562]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:33:56.434783 ip-10-0-139-17 kubenswrapper[2562]: , stdout: , stderr: , exit code -1 Sep 16 19:33:56.434783 ip-10-0-139-17 kubenswrapper[2562]: > containerID="f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518" cmd=["/bin/sleep","15"] Sep 16 19:33:56.435302 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:56.434810 2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" message="" err=< Sep 16 19:33:56.435302 ip-10-0-139-17 kubenswrapper[2562]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:33:56.435302 ip-10-0-139-17 kubenswrapper[2562]: , stdout: , stderr: , exit code -1 Sep 16 19:33:56.435302 ip-10-0-139-17 kubenswrapper[2562]: > Sep 16 19:33:56.435302 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:33:56.434827 2562 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:33:56.435302 ip-10-0-139-17 kubenswrapper[2562]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:33:56.435302 ip-10-0-139-17 kubenswrapper[2562]: , stdout: , stderr: , exit code -1 Sep 16 19:33:56.435302 ip-10-0-139-17 kubenswrapper[2562]: > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="main" containerID="cri-o://f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518" Sep 16 19:33:56.435302 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:56.434924 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="main" containerID="cri-o://f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518" gracePeriod=60 Sep 16 19:33:56.475744 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:56.475655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:56.653233 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:56.653154 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="llm-d-routing-sidecar" containerID="cri-o://9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265" gracePeriod=60 Sep 16 19:33:56.741374 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:56.741331 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.22:8000/health\": dial tcp 10.132.0.22:8000: connect: connection refused" Sep 16 19:33:56.975430 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:56.975358 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:33:57.106210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106116 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-model-cache\") pod \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " Sep 16 19:33:57.106210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106202 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-dshm\") pod \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " Sep 16 19:33:57.106697 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106307 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-home\") pod \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " Sep 16 19:33:57.106697 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106388 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tls-certs\") pod \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " Sep 16 19:33:57.106697 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106431 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4lpv\" (UniqueName: \"kubernetes.io/projected/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-kube-api-access-n4lpv\") pod \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " Sep 16 19:33:57.106697 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106468 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tmp-dir\") pod \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\" (UID: \"d3cc27c1-a832-49a7-9dec-3dff51cba1cb\") " Sep 16 19:33:57.106927 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106712 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-home" (OuterVolumeSpecName: "home") pod "d3cc27c1-a832-49a7-9dec-3dff51cba1cb" (UID: "d3cc27c1-a832-49a7-9dec-3dff51cba1cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:33:57.106927 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106728 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3cc27c1-a832-49a7-9dec-3dff51cba1cb" (UID: "d3cc27c1-a832-49a7-9dec-3dff51cba1cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:33:57.106927 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106858 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3cc27c1-a832-49a7-9dec-3dff51cba1cb" (UID: "d3cc27c1-a832-49a7-9dec-3dff51cba1cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:33:57.107120 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106933 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:33:57.107120 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.106949 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:33:57.110445 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.110369 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-dshm" (OuterVolumeSpecName: "dshm") pod "d3cc27c1-a832-49a7-9dec-3dff51cba1cb" (UID: "d3cc27c1-a832-49a7-9dec-3dff51cba1cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:33:57.110445 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.110355 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3cc27c1-a832-49a7-9dec-3dff51cba1cb" (UID: "d3cc27c1-a832-49a7-9dec-3dff51cba1cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:33:57.110771 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.110382 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-kube-api-access-n4lpv" (OuterVolumeSpecName: "kube-api-access-n4lpv") pod "d3cc27c1-a832-49a7-9dec-3dff51cba1cb" (UID: "d3cc27c1-a832-49a7-9dec-3dff51cba1cb"). InnerVolumeSpecName "kube-api-access-n4lpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:33:57.208001 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.207854 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:33:57.208001 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.207898 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4lpv\" (UniqueName: \"kubernetes.io/projected/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-kube-api-access-n4lpv\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:33:57.208001 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.207909 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:33:57.208001 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.207925 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cc27c1-a832-49a7-9dec-3dff51cba1cb-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:33:57.384610 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.384579 2562 generic.go:356] "Generic (PLEG): container finished" podID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerID="f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518" exitCode=0 Sep 16 19:33:57.384610 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.384602 2562 generic.go:356] "Generic (PLEG): container finished" podID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerID="9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265" exitCode=0 Sep 16 19:33:57.384807 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.384630 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" Sep 16 19:33:57.384807 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.384663 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" event={"ID":"d3cc27c1-a832-49a7-9dec-3dff51cba1cb","Type":"ContainerDied","Data":"f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518"} Sep 16 19:33:57.384807 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.384718 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" event={"ID":"d3cc27c1-a832-49a7-9dec-3dff51cba1cb","Type":"ContainerDied","Data":"9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265"} Sep 16 19:33:57.384807 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.384737 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" event={"ID":"d3cc27c1-a832-49a7-9dec-3dff51cba1cb","Type":"ContainerDied","Data":"b3877fac7cf44d9a79dd0560c92cd7eac566537a6999182f3816f41c44612524"} Sep 16 19:33:57.384807 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.384760 2562 scope.go:117] "RemoveContainer" containerID="f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518" Sep 16 19:33:57.393595 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.393578 2562 scope.go:117] "RemoveContainer" containerID="9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265" Sep 16 19:33:57.401343 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.401325 2562 scope.go:117] "RemoveContainer" containerID="f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518" Sep 16 19:33:57.401621 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:33:57.401604 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518\": container with ID starting with f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518 not found: ID does not exist" containerID="f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518" Sep 16 19:33:57.401675 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.401630 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518"} err="failed to get container status \"f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518\": rpc error: code = NotFound desc = could not find container \"f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518\": container with ID starting with f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518 not found: ID does not exist" Sep 16 19:33:57.401675 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.401646 2562 scope.go:117] "RemoveContainer" containerID="9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265" Sep 16 19:33:57.401897 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:33:57.401880 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265\": container with ID starting with 9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265 not found: ID does not exist" containerID="9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265" Sep 16 19:33:57.401941 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.401903 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265"} err="failed to get container status \"9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265\": rpc error: code = NotFound desc = could not find container \"9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265\": container with ID starting with 9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265 not found: ID does not exist" Sep 16 19:33:57.401941 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.401920 2562 scope.go:117] "RemoveContainer" containerID="f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518" Sep 16 19:33:57.402175 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.402157 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518"} err="failed to get container status \"f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518\": rpc error: code = NotFound desc = could not find container \"f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518\": container with ID starting with f84e7dda61db38769f4d1985f291e8906b2a9c16bfa5f4db4e3e9e40829ad518 not found: ID does not exist" Sep 16 19:33:57.402253 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.402177 2562 scope.go:117] "RemoveContainer" containerID="9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265" Sep 16 19:33:57.402387 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.402372 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265"} err="failed to get container status \"9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265\": rpc error: code = NotFound desc = could not find container \"9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265\": container with ID starting with 9c752738fc6eae2cda47e0e69f6c47afc51ad916666e4d4bd04e9d3f49397265 not found: ID does not exist" Sep 16 19:33:57.446548 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.446520 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk"] Sep 16 19:33:57.460299 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.460247 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk"] Sep 16 19:33:57.887411 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.887357 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-lkbfk" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.22:8001/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 16 19:33:57.962470 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:33:57.962438 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" path="/var/lib/kubelet/pods/d3cc27c1-a832-49a7-9dec-3dff51cba1cb/volumes" Sep 16 19:34:42.569695 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:34:42.569657 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:42.700496 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:34:42.700443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:50.455233 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:34:50.455132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:51.979731 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:34:51.979706 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:34:51.983094 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:34:51.983074 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:34:56.744739 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:34:56.744701 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:56.951672 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:34:56.951637 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:57.186210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:34:57.186174 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:57.286903 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:34:57.286872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:04.542232 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:04.542201 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:07.424877 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:07.424850 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:23.648452 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:23.648398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:32.078617 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:32.078578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:32.203619 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:32.203591 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:42.778462 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:42.778427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:42.925205 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:42.925171 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:43.059497 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:43.059417 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:43.148272 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:43.148244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:35:47.815935 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:35:47.815901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:03.077395 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:03.077361 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:14.112914 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:14.112879 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:22.015670 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:22.015616 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:22.112646 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:22.112608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:26.112612 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:26.112577 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:29.091196 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:29.091145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:32.439184 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:32.439153 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:32.521769 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:32.521739 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:35.053757 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:35.053724 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:35.167540 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:35.167509 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:41.329090 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:41.329056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:41.588110 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:41.588076 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:41.915971 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:41.915888 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:56.744831 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:56.744796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:36:59.911548 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:36:59.911510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:37:00.023206 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:00.023157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:37:00.115200 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:00.115171 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:37:02.693360 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:02.693251 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:37:04.052408 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:04.052382 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:37:05.101098 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:05.101070 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:37:14.666981 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.666946 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg"] Sep 16 19:37:14.667455 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.667212 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="llm-d-routing-sidecar" Sep 16 19:37:14.667455 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.667224 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="llm-d-routing-sidecar" Sep 16 19:37:14.667455 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.667241 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="main" Sep 16 19:37:14.667455 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.667247 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="main" Sep 16 19:37:14.667455 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.667303 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="llm-d-routing-sidecar" Sep 16 19:37:14.667455 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.667312 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3cc27c1-a832-49a7-9dec-3dff51cba1cb" containerName="main" Sep 16 19:37:14.669945 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.669931 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.672906 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.672881 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\"" Sep 16 19:37:14.673046 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.672889 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-mtx7k\"" Sep 16 19:37:14.673555 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.673538 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\"" Sep 16 19:37:14.673555 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.673549 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\"" Sep 16 19:37:14.686795 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.686768 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg"] Sep 16 19:37:14.741270 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.741236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:37:14.779562 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.779536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-home\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.779715 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.779575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tls-certs\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.779715 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.779605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-model-cache\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.779715 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.779626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5svpr\" (UniqueName: \"kubernetes.io/projected/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-kube-api-access-5svpr\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.779715 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.779664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-dshm\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.779715 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.779699 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.880469 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.880427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-dshm\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.880651 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.880480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.880651 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.880507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-home\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.880651 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.880542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tls-certs\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.880651 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.880577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-model-cache\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.880651 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.880602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5svpr\" (UniqueName: \"kubernetes.io/projected/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-kube-api-access-5svpr\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.881047 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.881001 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-home\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.881141 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.881001 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.881141 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.881071 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-model-cache\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.882855 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.882831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-dshm\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.882997 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.882980 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tls-certs\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.893662 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.893631 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5svpr\" (UniqueName: \"kubernetes.io/projected/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-kube-api-access-5svpr\") pod \"test-div-v1-kserve-fb566d667-l57cg\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:14.979640 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:14.979530 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:15.250191 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:15.250123 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg"] Sep 16 19:37:15.252691 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:37:15.252663 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6b15b8_e9b1_48e3_a4d2_10c1cdcc7d6c.slice/crio-5863615f410d8984f07c37fad08387e61ec8818150df31af0a96dfd5075e7e92 WatchSource:0}: Error finding container 5863615f410d8984f07c37fad08387e61ec8818150df31af0a96dfd5075e7e92: Status 404 returned error can't find the container with id 5863615f410d8984f07c37fad08387e61ec8818150df31af0a96dfd5075e7e92 Sep 16 19:37:15.254495 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:15.254480 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:37:16.115846 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:16.115804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" event={"ID":"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c","Type":"ContainerStarted","Data":"5863615f410d8984f07c37fad08387e61ec8818150df31af0a96dfd5075e7e92"} Sep 16 19:37:17.121856 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:17.121825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" event={"ID":"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c","Type":"ContainerStarted","Data":"0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac"} Sep 16 19:37:17.148361 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:17.148311 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" podStartSLOduration=2.051538442 podStartE2EDuration="3.148293046s" totalImagesPullingTime="1.096754604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:17.147977035 +0000 UTC m=+1945.697034322" watchObservedRunningTime="2026-09-16 19:37:17.148293046 +0000 UTC m=+1945.697350321" Sep 16 19:37:17.980549 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:17.980505 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:24.980195 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:24.980150 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:24.992633 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:24.992604 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:25.162424 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.162396 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:37:25.695019 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.694980 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq"] Sep 16 19:37:25.697888 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.697871 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.703357 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.703335 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\"" Sep 16 19:37:25.726843 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.726800 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq"] Sep 16 19:37:25.771001 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.770967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdg5\" (UniqueName: \"kubernetes.io/projected/6037aa45-e3b6-4b3e-9849-b439aee272c3-kube-api-access-dmdg5\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.771001 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.771005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.771216 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.771107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-home\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.771216 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.771139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6037aa45-e3b6-4b3e-9849-b439aee272c3-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.771216 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.771170 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.771216 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.771187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.788360 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.788329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:37:25.871974 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.871935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdg5\" (UniqueName: \"kubernetes.io/projected/6037aa45-e3b6-4b3e-9849-b439aee272c3-kube-api-access-dmdg5\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.871974 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.871978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.872204 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.872063 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-home\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.872204 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.872097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6037aa45-e3b6-4b3e-9849-b439aee272c3-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.872204 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.872143 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.872204 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.872168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.872506 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.872483 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-home\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.872579 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.872506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.872579 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.872530 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.874258 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.874239 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.874598 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.874577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6037aa45-e3b6-4b3e-9849-b439aee272c3-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:25.884222 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:25.884196 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdg5\" (UniqueName: \"kubernetes.io/projected/6037aa45-e3b6-4b3e-9849-b439aee272c3-kube-api-access-dmdg5\") pod \"test-div-v2-kserve-5f6fdb6b77-4qvvq\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:26.007382 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:26.007290 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:26.280187 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:26.280003 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq"] Sep 16 19:37:26.282659 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:37:26.282626 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6037aa45_e3b6_4b3e_9849_b439aee272c3.slice/crio-fa3f56e008ba0607d67659b898c5c90d2a5a638c803bc43e7a0380837f308adb WatchSource:0}: Error finding container fa3f56e008ba0607d67659b898c5c90d2a5a638c803bc43e7a0380837f308adb: Status 404 returned error can't find the container with id fa3f56e008ba0607d67659b898c5c90d2a5a638c803bc43e7a0380837f308adb Sep 16 19:37:27.161647 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:27.161612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" event={"ID":"6037aa45-e3b6-4b3e-9849-b439aee272c3","Type":"ContainerStarted","Data":"cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0"} Sep 16 19:37:27.162158 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:27.161665 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" event={"ID":"6037aa45-e3b6-4b3e-9849-b439aee272c3","Type":"ContainerStarted","Data":"fa3f56e008ba0607d67659b898c5c90d2a5a638c803bc43e7a0380837f308adb"} Sep 16 19:37:27.202963 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:27.202912 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" podStartSLOduration=2.202896614 podStartE2EDuration="2.202896614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:27.202106808 +0000 UTC m=+1955.751164081" watchObservedRunningTime="2026-09-16 19:37:27.202896614 +0000 UTC m=+1955.751953942" Sep 16 19:37:28.008270 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:28.008238 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:36.008333 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:36.008299 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:36.020703 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:36.020671 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:36.212129 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:36.212100 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:37:56.450190 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:37:56.450105 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:02.730481 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:02.730433 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:02.830757 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:02.830720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:10.975836 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:10.975796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:11.088928 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:11.088899 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:11.195500 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:11.195460 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:11.291043 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:11.290946 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq"] Sep 16 19:38:11.359633 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:11.359602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:11.466655 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:11.466621 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:11.569166 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:11.569079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:11.682553 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:11.682523 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg"] Sep 16 19:38:11.729158 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:11.729125 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:16.047327 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:16.047290 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:18.305770 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:18.305708 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:20.811244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:20.811210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:24.401215 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:24.401182 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:24.482585 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:24.482550 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:26.312223 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.312182 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" podUID="6037aa45-e3b6-4b3e-9849-b439aee272c3" containerName="main" containerID="cri-o://cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0" gracePeriod=45 Sep 16 19:38:26.564856 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.564798 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:38:26.636960 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.636921 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6037aa45-e3b6-4b3e-9849-b439aee272c3-tls-certs\") pod \"6037aa45-e3b6-4b3e-9849-b439aee272c3\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " Sep 16 19:38:26.637145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.636977 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-model-cache\") pod \"6037aa45-e3b6-4b3e-9849-b439aee272c3\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " Sep 16 19:38:26.637145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.637001 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmdg5\" (UniqueName: \"kubernetes.io/projected/6037aa45-e3b6-4b3e-9849-b439aee272c3-kube-api-access-dmdg5\") pod \"6037aa45-e3b6-4b3e-9849-b439aee272c3\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " Sep 16 19:38:26.637145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.637114 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-home\") pod \"6037aa45-e3b6-4b3e-9849-b439aee272c3\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " Sep 16 19:38:26.637269 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.637166 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-tmp-dir\") pod \"6037aa45-e3b6-4b3e-9849-b439aee272c3\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " Sep 16 19:38:26.637269 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.637213 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-dshm\") pod \"6037aa45-e3b6-4b3e-9849-b439aee272c3\" (UID: \"6037aa45-e3b6-4b3e-9849-b439aee272c3\") " Sep 16 19:38:26.637371 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.637283 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "6037aa45-e3b6-4b3e-9849-b439aee272c3" (UID: "6037aa45-e3b6-4b3e-9849-b439aee272c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:26.637495 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.637401 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-home" (OuterVolumeSpecName: "home") pod "6037aa45-e3b6-4b3e-9849-b439aee272c3" (UID: "6037aa45-e3b6-4b3e-9849-b439aee272c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:26.637495 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.637442 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:26.637495 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.637457 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6037aa45-e3b6-4b3e-9849-b439aee272c3" (UID: "6037aa45-e3b6-4b3e-9849-b439aee272c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:26.639290 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.639270 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6037aa45-e3b6-4b3e-9849-b439aee272c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6037aa45-e3b6-4b3e-9849-b439aee272c3" (UID: "6037aa45-e3b6-4b3e-9849-b439aee272c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:38:26.639368 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.639310 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-dshm" (OuterVolumeSpecName: "dshm") pod "6037aa45-e3b6-4b3e-9849-b439aee272c3" (UID: "6037aa45-e3b6-4b3e-9849-b439aee272c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:26.639368 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.639343 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6037aa45-e3b6-4b3e-9849-b439aee272c3-kube-api-access-dmdg5" (OuterVolumeSpecName: "kube-api-access-dmdg5") pod "6037aa45-e3b6-4b3e-9849-b439aee272c3" (UID: "6037aa45-e3b6-4b3e-9849-b439aee272c3"). InnerVolumeSpecName "kube-api-access-dmdg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:38:26.701027 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.700969 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" podUID="0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" containerName="main" containerID="cri-o://0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac" gracePeriod=45 Sep 16 19:38:26.737860 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.737823 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdg5\" (UniqueName: \"kubernetes.io/projected/6037aa45-e3b6-4b3e-9849-b439aee272c3-kube-api-access-dmdg5\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:26.737860 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.737853 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:26.737860 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.737862 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:26.737860 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.737869 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6037aa45-e3b6-4b3e-9849-b439aee272c3-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:26.738129 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.737877 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6037aa45-e3b6-4b3e-9849-b439aee272c3-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:26.933285 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:26.933261 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:38:27.039703 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.039666 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5svpr\" (UniqueName: \"kubernetes.io/projected/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-kube-api-access-5svpr\") pod \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " Sep 16 19:38:27.039703 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.039709 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-dshm\") pod \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " Sep 16 19:38:27.039949 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.039738 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-home\") pod \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " Sep 16 19:38:27.039949 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.039753 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tmp-dir\") pod \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " Sep 16 19:38:27.039949 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.039798 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-model-cache\") pod \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " Sep 16 19:38:27.039949 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.039825 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tls-certs\") pod \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\" (UID: \"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c\") " Sep 16 19:38:27.040184 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.040044 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" (UID: "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:27.040184 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.040077 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" (UID: "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:27.040184 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.040093 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-home" (OuterVolumeSpecName: "home") pod "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" (UID: "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:27.040184 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.040150 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:27.040184 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.040167 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:27.040184 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.040179 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:27.041945 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.041924 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-kube-api-access-5svpr" (OuterVolumeSpecName: "kube-api-access-5svpr") pod "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" (UID: "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c"). InnerVolumeSpecName "kube-api-access-5svpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:38:27.041945 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.041929 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-dshm" (OuterVolumeSpecName: "dshm") pod "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" (UID: "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:27.042062 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.041978 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" (UID: "0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:38:27.141339 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.141290 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:27.141339 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.141337 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5svpr\" (UniqueName: \"kubernetes.io/projected/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-kube-api-access-5svpr\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:27.141339 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.141347 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:38:27.405401 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.405319 2562 generic.go:356] "Generic (PLEG): container finished" podID="0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" containerID="0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac" exitCode=0 Sep 16 19:38:27.405401 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.405365 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" Sep 16 19:38:27.405401 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.405389 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" event={"ID":"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c","Type":"ContainerDied","Data":"0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac"} Sep 16 19:38:27.405903 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.405441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg" event={"ID":"0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c","Type":"ContainerDied","Data":"5863615f410d8984f07c37fad08387e61ec8818150df31af0a96dfd5075e7e92"} Sep 16 19:38:27.405903 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.405467 2562 scope.go:117] "RemoveContainer" containerID="0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac" Sep 16 19:38:27.406879 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.406657 2562 generic.go:356] "Generic (PLEG): container finished" podID="6037aa45-e3b6-4b3e-9849-b439aee272c3" containerID="cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0" exitCode=0 Sep 16 19:38:27.406879 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.406708 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" Sep 16 19:38:27.406879 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.406716 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" event={"ID":"6037aa45-e3b6-4b3e-9849-b439aee272c3","Type":"ContainerDied","Data":"cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0"} Sep 16 19:38:27.406879 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.406816 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq" event={"ID":"6037aa45-e3b6-4b3e-9849-b439aee272c3","Type":"ContainerDied","Data":"fa3f56e008ba0607d67659b898c5c90d2a5a638c803bc43e7a0380837f308adb"} Sep 16 19:38:27.418452 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.413892 2562 scope.go:117] "RemoveContainer" containerID="0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac" Sep 16 19:38:27.418452 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:38:27.414297 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac\": container with ID starting with 0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac not found: ID does not exist" containerID="0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac" Sep 16 19:38:27.418452 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.414339 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac"} err="failed to get container status \"0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac\": rpc error: code = NotFound desc = could not find container \"0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac\": container with ID starting with 0636dc13140964fb60a9ad915b9c3eaacce0544e0bdc14831d124fc049a516ac not found: ID does not exist" Sep 16 19:38:27.418452 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.414364 2562 scope.go:117] "RemoveContainer" containerID="cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0" Sep 16 19:38:27.425831 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.425812 2562 scope.go:117] "RemoveContainer" containerID="cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0" Sep 16 19:38:27.426113 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:38:27.426092 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0\": container with ID starting with cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0 not found: ID does not exist" containerID="cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0" Sep 16 19:38:27.426204 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.426118 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0"} err="failed to get container status \"cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0\": rpc error: code = NotFound desc = could not find container \"cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0\": container with ID starting with cff3a0b187988aaa6c2851124f15b595a71c9355283b07d5ecd15f79a0ca40d0 not found: ID does not exist" Sep 16 19:38:27.438504 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.438481 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq"] Sep 16 19:38:27.444952 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.444930 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4qvvq"] Sep 16 19:38:27.459157 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.459134 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg"] Sep 16 19:38:27.465265 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.465240 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-l57cg"] Sep 16 19:38:27.962331 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.962299 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" path="/var/lib/kubelet/pods/0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c/volumes" Sep 16 19:38:27.962676 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:27.962663 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6037aa45-e3b6-4b3e-9849-b439aee272c3" path="/var/lib/kubelet/pods/6037aa45-e3b6-4b3e-9849-b439aee272c3/volumes" Sep 16 19:38:30.050216 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.050184 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm"] Sep 16 19:38:30.050706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.050560 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6037aa45-e3b6-4b3e-9849-b439aee272c3" containerName="main" Sep 16 19:38:30.050706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.050577 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6037aa45-e3b6-4b3e-9849-b439aee272c3" containerName="main" Sep 16 19:38:30.050706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.050599 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" containerName="main" Sep 16 19:38:30.050706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.050607 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" containerName="main" Sep 16 19:38:30.050706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.050696 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6b15b8-e9b1-48e3-a4d2-10c1cdcc7d6c" containerName="main" Sep 16 19:38:30.050706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.050707 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6037aa45-e3b6-4b3e-9849-b439aee272c3" containerName="main" Sep 16 19:38:30.055088 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.055067 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.058877 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.058857 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\"" Sep 16 19:38:30.059003 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.058865 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-7twkp\"" Sep 16 19:38:30.059101 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.059039 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\"" Sep 16 19:38:30.059101 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.059080 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\"" Sep 16 19:38:30.067436 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.067414 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm"] Sep 16 19:38:30.095761 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.095728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:30.163556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.163517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-home\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.163556 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.163554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.163766 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.163571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5f2t\" (UniqueName: \"kubernetes.io/projected/deb86972-6541-44cb-b524-1bcec3baf51f-kube-api-access-l5f2t\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.163766 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.163637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-dshm\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.163766 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.163694 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-model-cache\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.163766 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.163713 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb86972-6541-44cb-b524-1bcec3baf51f-tls-certs\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.264706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.264651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-home\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.264706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.264707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.264934 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.264735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5f2t\" (UniqueName: \"kubernetes.io/projected/deb86972-6541-44cb-b524-1bcec3baf51f-kube-api-access-l5f2t\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.264934 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.264769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-dshm\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.264934 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.264796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-model-cache\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.265118 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.264924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb86972-6541-44cb-b524-1bcec3baf51f-tls-certs\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.265177 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.265156 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-home\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.265238 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.265181 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-model-cache\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.265316 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.265294 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.267116 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.267093 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-dshm\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.267423 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.267405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb86972-6541-44cb-b524-1bcec3baf51f-tls-certs\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.273653 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.273631 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5f2t\" (UniqueName: \"kubernetes.io/projected/deb86972-6541-44cb-b524-1bcec3baf51f-kube-api-access-l5f2t\") pod \"stop-v2-kserve-7f68747c45-9fpvm\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.365942 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.365910 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:30.645534 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:30.645506 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm"] Sep 16 19:38:30.648671 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:38:30.648640 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb86972_6541_44cb_b524_1bcec3baf51f.slice/crio-3bbe852ae70941e2d73141ede7e26de7b5ce8acbdbf62c695902bcd30b2ef245 WatchSource:0}: Error finding container 3bbe852ae70941e2d73141ede7e26de7b5ce8acbdbf62c695902bcd30b2ef245: Status 404 returned error can't find the container with id 3bbe852ae70941e2d73141ede7e26de7b5ce8acbdbf62c695902bcd30b2ef245 Sep 16 19:38:31.008073 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:31.007979 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:31.425637 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:31.425597 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" event={"ID":"deb86972-6541-44cb-b524-1bcec3baf51f","Type":"ContainerStarted","Data":"5120ece8700c8ba894a45a58e6130325292600501073c1f6b4a9348f360116c1"} Sep 16 19:38:31.426050 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:31.425657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" event={"ID":"deb86972-6541-44cb-b524-1bcec3baf51f","Type":"ContainerStarted","Data":"3bbe852ae70941e2d73141ede7e26de7b5ce8acbdbf62c695902bcd30b2ef245"} Sep 16 19:38:31.446998 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:31.446938 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" podStartSLOduration=1.446922047 podStartE2EDuration="1.446922047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:31.445296394 +0000 UTC m=+2019.994353669" watchObservedRunningTime="2026-09-16 19:38:31.446922047 +0000 UTC m=+2019.995979320" Sep 16 19:38:32.366028 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:32.365979 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:40.366753 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:40.366717 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:40.379169 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:40.379136 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:40.474567 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:40.474538 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:38:48.234657 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:48.234624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:48.333148 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:48.333096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:53.827893 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:53.827860 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:59.910561 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:59.910524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:38:59.994403 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:38:59.994370 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:03.429409 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:03.429377 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm"] Sep 16 19:39:03.465946 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:03.465917 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:18.448815 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.448712 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" podUID="deb86972-6541-44cb-b524-1bcec3baf51f" containerName="main" containerID="cri-o://5120ece8700c8ba894a45a58e6130325292600501073c1f6b4a9348f360116c1" gracePeriod=45 Sep 16 19:39:18.463704 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.463670 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" podUID="deb86972-6541-44cb-b524-1bcec3baf51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.25:8000/health\": dial tcp 10.132.0.25:8000: connect: connection refused" Sep 16 19:39:18.604270 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.604237 2562 generic.go:356] "Generic (PLEG): container finished" podID="deb86972-6541-44cb-b524-1bcec3baf51f" containerID="5120ece8700c8ba894a45a58e6130325292600501073c1f6b4a9348f360116c1" exitCode=0 Sep 16 19:39:18.604419 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.604311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" event={"ID":"deb86972-6541-44cb-b524-1bcec3baf51f","Type":"ContainerDied","Data":"5120ece8700c8ba894a45a58e6130325292600501073c1f6b4a9348f360116c1"} Sep 16 19:39:18.685887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.685863 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:39:18.755664 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.755575 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-model-cache\") pod \"deb86972-6541-44cb-b524-1bcec3baf51f\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " Sep 16 19:39:18.755664 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.755613 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-tmp-dir\") pod \"deb86972-6541-44cb-b524-1bcec3baf51f\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " Sep 16 19:39:18.755664 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.755636 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-dshm\") pod \"deb86972-6541-44cb-b524-1bcec3baf51f\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " Sep 16 19:39:18.755664 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.755663 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5f2t\" (UniqueName: \"kubernetes.io/projected/deb86972-6541-44cb-b524-1bcec3baf51f-kube-api-access-l5f2t\") pod \"deb86972-6541-44cb-b524-1bcec3baf51f\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " Sep 16 19:39:18.755971 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.755887 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "deb86972-6541-44cb-b524-1bcec3baf51f" (UID: "deb86972-6541-44cb-b524-1bcec3baf51f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:39:18.755971 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.755903 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-model-cache" (OuterVolumeSpecName: "model-cache") pod "deb86972-6541-44cb-b524-1bcec3baf51f" (UID: "deb86972-6541-44cb-b524-1bcec3baf51f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:39:18.757855 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.757833 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-dshm" (OuterVolumeSpecName: "dshm") pod "deb86972-6541-44cb-b524-1bcec3baf51f" (UID: "deb86972-6541-44cb-b524-1bcec3baf51f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:39:18.757942 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.757924 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb86972-6541-44cb-b524-1bcec3baf51f-kube-api-access-l5f2t" (OuterVolumeSpecName: "kube-api-access-l5f2t") pod "deb86972-6541-44cb-b524-1bcec3baf51f" (UID: "deb86972-6541-44cb-b524-1bcec3baf51f"). InnerVolumeSpecName "kube-api-access-l5f2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:39:18.856328 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.856287 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-home\") pod \"deb86972-6541-44cb-b524-1bcec3baf51f\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " Sep 16 19:39:18.856510 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.856381 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb86972-6541-44cb-b524-1bcec3baf51f-tls-certs\") pod \"deb86972-6541-44cb-b524-1bcec3baf51f\" (UID: \"deb86972-6541-44cb-b524-1bcec3baf51f\") " Sep 16 19:39:18.856510 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.856490 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:39:18.856624 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.856520 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:39:18.856624 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.856528 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:39:18.856624 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.856536 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5f2t\" (UniqueName: \"kubernetes.io/projected/deb86972-6541-44cb-b524-1bcec3baf51f-kube-api-access-l5f2t\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:39:18.856624 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.856576 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-home" (OuterVolumeSpecName: "home") pod "deb86972-6541-44cb-b524-1bcec3baf51f" (UID: "deb86972-6541-44cb-b524-1bcec3baf51f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:39:18.858372 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.858352 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb86972-6541-44cb-b524-1bcec3baf51f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "deb86972-6541-44cb-b524-1bcec3baf51f" (UID: "deb86972-6541-44cb-b524-1bcec3baf51f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:39:18.957512 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.957463 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb86972-6541-44cb-b524-1bcec3baf51f-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:39:18.957512 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:18.957498 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb86972-6541-44cb-b524-1bcec3baf51f-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:39:19.609394 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:19.609358 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" event={"ID":"deb86972-6541-44cb-b524-1bcec3baf51f","Type":"ContainerDied","Data":"3bbe852ae70941e2d73141ede7e26de7b5ce8acbdbf62c695902bcd30b2ef245"} Sep 16 19:39:19.609394 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:19.609378 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm" Sep 16 19:39:19.609852 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:19.609428 2562 scope.go:117] "RemoveContainer" containerID="5120ece8700c8ba894a45a58e6130325292600501073c1f6b4a9348f360116c1" Sep 16 19:39:19.631805 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:19.631777 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm"] Sep 16 19:39:19.636997 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:19.636973 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9fpvm"] Sep 16 19:39:19.962233 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:19.962200 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb86972-6541-44cb-b524-1bcec3baf51f" path="/var/lib/kubelet/pods/deb86972-6541-44cb-b524-1bcec3baf51f/volumes" Sep 16 19:39:25.167892 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:25.167857 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:25.272242 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:25.272209 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:25.374615 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:25.374582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:30.233899 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:30.233829 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:36.083390 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:36.083358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:36.205109 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:36.205079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:42.665716 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:42.665681 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:48.668875 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:48.668840 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:48.737600 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:48.737565 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:39:52.002381 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:52.002349 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:39:52.006320 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:39:52.006294 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:40:02.969907 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:40:02.969874 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:40:08.427290 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:40:08.427245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:40:14.390600 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:40:14.390544 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:40:14.491314 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:40:14.491282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:40:31.149267 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:40:31.149229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:40:39.617355 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:40:39.617319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:40:39.913927 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:40:39.913843 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:40:40.003300 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:40:40.003249 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:40:59.546958 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:40:59.546876 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:41:18.593279 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:41:18.593244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:41:18.685559 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:41:18.685495 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:41:19.480400 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:41:19.480371 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:41:35.572617 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:41:35.572562 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:41:35.686342 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:41:35.686308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:41:35.784804 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:41:35.784774 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:41:40.634834 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:41:40.634800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:41:51.348520 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:41:51.348482 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:06.698991 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:06.698960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:06.799330 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:06.799296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:13.775431 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:13.775395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:18.582979 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:18.582904 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:20.035991 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:20.035956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:24.651777 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:24.651744 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:24.751992 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:24.751957 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:32.615666 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:32.615635 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:45.877372 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:45.877336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:53.504233 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:53.504196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:53.761332 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:53.761255 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:42:53.857312 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:42:53.857279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:00.884847 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:00.884811 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:11.557884 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:11.557842 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:11.674738 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:11.674704 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:11.789005 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:11.788964 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:11.894260 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:11.894224 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:12.543826 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:12.543789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:13.280545 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:13.280514 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:16.582286 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:16.582251 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:21.761851 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.761813 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz"] Sep 16 19:43:21.762256 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.762093 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb86972-6541-44cb-b524-1bcec3baf51f" containerName="main" Sep 16 19:43:21.762256 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.762106 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb86972-6541-44cb-b524-1bcec3baf51f" containerName="main" Sep 16 19:43:21.762256 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.762161 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb86972-6541-44cb-b524-1bcec3baf51f" containerName="main" Sep 16 19:43:21.764962 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.764946 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.767807 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.767781 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\"" Sep 16 19:43:21.768037 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.768024 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-xjmqm\"" Sep 16 19:43:21.768829 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.768811 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\"" Sep 16 19:43:21.768924 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.768891 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\"" Sep 16 19:43:21.780547 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.780523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz"] Sep 16 19:43:21.848594 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.848556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:21.896567 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.896529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-home\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.896750 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.896577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.896750 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.896596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-dshm\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.896750 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.896637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.896750 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.896665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d691a222-56c1-4d4c-8eec-7480e3d154ec-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.896750 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.896680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6dkb\" (UniqueName: \"kubernetes.io/projected/d691a222-56c1-4d4c-8eec-7480e3d154ec-kube-api-access-q6dkb\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.997081 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.997043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.997249 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.997102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d691a222-56c1-4d4c-8eec-7480e3d154ec-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.997249 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.997129 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6dkb\" (UniqueName: \"kubernetes.io/projected/d691a222-56c1-4d4c-8eec-7480e3d154ec-kube-api-access-q6dkb\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.997249 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.997153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-home\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.997249 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.997183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.997447 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.997308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-dshm\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.997700 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.997503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.997700 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.997519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.997700 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.997574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-home\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.999403 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.999382 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-dshm\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:21.999730 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:21.999712 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d691a222-56c1-4d4c-8eec-7480e3d154ec-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:22.007359 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:22.007334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6dkb\" (UniqueName: \"kubernetes.io/projected/d691a222-56c1-4d4c-8eec-7480e3d154ec-kube-api-access-q6dkb\") pod \"tri-v1-kserve-79f8f98bc5-wbfsz\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:22.074998 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:22.074907 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:22.367706 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:22.367673 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz"] Sep 16 19:43:22.371228 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:43:22.371201 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd691a222_56c1_4d4c_8eec_7480e3d154ec.slice/crio-4c00222087cfe48cef0a9c11e163efedf90006c69d8feeeff08122b0f6106454 WatchSource:0}: Error finding container 4c00222087cfe48cef0a9c11e163efedf90006c69d8feeeff08122b0f6106454: Status 404 returned error can't find the container with id 4c00222087cfe48cef0a9c11e163efedf90006c69d8feeeff08122b0f6106454 Sep 16 19:43:22.502183 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:22.502147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" event={"ID":"d691a222-56c1-4d4c-8eec-7480e3d154ec","Type":"ContainerStarted","Data":"bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d"} Sep 16 19:43:22.502355 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:22.502199 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" event={"ID":"d691a222-56c1-4d4c-8eec-7480e3d154ec","Type":"ContainerStarted","Data":"4c00222087cfe48cef0a9c11e163efedf90006c69d8feeeff08122b0f6106454"} Sep 16 19:43:22.528133 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:22.528080 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" podStartSLOduration=1.5280645430000002 podStartE2EDuration="1.528064543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:22.526900697 +0000 UTC m=+2311.075957972" watchObservedRunningTime="2026-09-16 19:43:22.528064543 +0000 UTC m=+2311.077121816" Sep 16 19:43:23.075712 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:23.075672 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:32.075680 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:32.075628 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:32.088295 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:32.088266 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:32.549649 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:32.549621 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:43:38.040885 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.040846 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz"] Sep 16 19:43:38.044051 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.044029 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.046614 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.046597 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\"" Sep 16 19:43:38.059723 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.059696 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz"] Sep 16 19:43:38.096745 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.096711 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:38.118145 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.118117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12722cfa-83e8-4c10-a373-81997319aa4e-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.118297 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.118148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hb8x\" (UniqueName: \"kubernetes.io/projected/12722cfa-83e8-4c10-a373-81997319aa4e-kube-api-access-4hb8x\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.118297 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.118175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.118297 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.118230 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-dshm\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.118297 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.118283 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-model-cache\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.118429 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.118315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-home\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.218933 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.218896 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-model-cache\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.218933 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.218931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-home\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.219203 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.218971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12722cfa-83e8-4c10-a373-81997319aa4e-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.219203 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.218988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hb8x\" (UniqueName: \"kubernetes.io/projected/12722cfa-83e8-4c10-a373-81997319aa4e-kube-api-access-4hb8x\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.219203 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.219035 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.219203 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.219189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-dshm\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.219422 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.219347 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-home\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.219422 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.219393 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.219529 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.219451 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-model-cache\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.221165 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.221144 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-dshm\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.221522 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.221506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12722cfa-83e8-4c10-a373-81997319aa4e-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.227940 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.227922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hb8x\" (UniqueName: \"kubernetes.io/projected/12722cfa-83e8-4c10-a373-81997319aa4e-kube-api-access-4hb8x\") pod \"tri-v2-kserve-dcb554bf5-dxrfz\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.357667 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.357630 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:38.630619 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:38.630530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz"] Sep 16 19:43:38.636812 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:43:38.635795 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12722cfa_83e8_4c10_a373_81997319aa4e.slice/crio-0a79c19f6b6b5aaff70273ed53d24db968f83c1310dbbb3bdda033800b5405cc WatchSource:0}: Error finding container 0a79c19f6b6b5aaff70273ed53d24db968f83c1310dbbb3bdda033800b5405cc: Status 404 returned error can't find the container with id 0a79c19f6b6b5aaff70273ed53d24db968f83c1310dbbb3bdda033800b5405cc Sep 16 19:43:39.565756 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:39.565721 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" event={"ID":"12722cfa-83e8-4c10-a373-81997319aa4e","Type":"ContainerStarted","Data":"48f485cba2f67d4c20c4592d99473ccbae9c67c864b7e3a364c247d336beba7b"} Sep 16 19:43:39.566122 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:39.565769 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" event={"ID":"12722cfa-83e8-4c10-a373-81997319aa4e","Type":"ContainerStarted","Data":"0a79c19f6b6b5aaff70273ed53d24db968f83c1310dbbb3bdda033800b5405cc"} Sep 16 19:43:39.589293 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:39.589243 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" podStartSLOduration=1.589226778 podStartE2EDuration="1.589226778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:39.588362767 +0000 UTC m=+2328.137420041" watchObservedRunningTime="2026-09-16 19:43:39.589226778 +0000 UTC m=+2328.138284053" Sep 16 19:43:40.358652 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:40.358611 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:48.358350 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:48.358251 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:48.370632 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:48.370605 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:48.611510 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:48.611415 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:43:49.131515 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:49.131477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:54.113437 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:54.113396 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:43:56.270217 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:43:56.270186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:44:00.286966 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:44:00.286927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:44:00.357328 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:44:00.357296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:44:01.254645 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:44:01.254607 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:44:39.282202 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:44:39.282152 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:44:39.360710 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:44:39.360678 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:44:52.025100 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:44:52.025069 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:44:52.028971 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:44:52.028946 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:44:52.371916 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:44:52.371879 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:25.969769 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:25.969695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:26.073613 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:26.073572 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:27.031868 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:27.031835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:27.143240 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:27.143204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:27.240672 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:27.240644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:27.893565 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:27.893533 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz"] Sep 16 19:45:27.962794 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:27.962759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:32.001685 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:31.997811 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz"] Sep 16 19:45:32.095853 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:32.095812 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:33.213228 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:33.213194 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:39.101907 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:39.101872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:39.178303 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:39.178256 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:42.912207 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:42.912162 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" podUID="12722cfa-83e8-4c10-a373-81997319aa4e" containerName="main" containerID="cri-o://48f485cba2f67d4c20c4592d99473ccbae9c67c864b7e3a364c247d336beba7b" gracePeriod=45 Sep 16 19:45:43.037921 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.037888 2562 generic.go:356] "Generic (PLEG): container finished" podID="12722cfa-83e8-4c10-a373-81997319aa4e" containerID="48f485cba2f67d4c20c4592d99473ccbae9c67c864b7e3a364c247d336beba7b" exitCode=0 Sep 16 19:45:43.038081 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.037959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" event={"ID":"12722cfa-83e8-4c10-a373-81997319aa4e","Type":"ContainerDied","Data":"48f485cba2f67d4c20c4592d99473ccbae9c67c864b7e3a364c247d336beba7b"} Sep 16 19:45:43.161649 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.161563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:43.161793 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.161734 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:45:43.315281 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315182 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-dshm\") pod \"12722cfa-83e8-4c10-a373-81997319aa4e\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " Sep 16 19:45:43.315281 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315243 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hb8x\" (UniqueName: \"kubernetes.io/projected/12722cfa-83e8-4c10-a373-81997319aa4e-kube-api-access-4hb8x\") pod \"12722cfa-83e8-4c10-a373-81997319aa4e\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " Sep 16 19:45:43.315281 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315277 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12722cfa-83e8-4c10-a373-81997319aa4e-tls-certs\") pod \"12722cfa-83e8-4c10-a373-81997319aa4e\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " Sep 16 19:45:43.315579 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315307 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-model-cache\") pod \"12722cfa-83e8-4c10-a373-81997319aa4e\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " Sep 16 19:45:43.315579 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315342 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-tmp-dir\") pod \"12722cfa-83e8-4c10-a373-81997319aa4e\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " Sep 16 19:45:43.315579 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315436 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-home\") pod \"12722cfa-83e8-4c10-a373-81997319aa4e\" (UID: \"12722cfa-83e8-4c10-a373-81997319aa4e\") " Sep 16 19:45:43.315720 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315613 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12722cfa-83e8-4c10-a373-81997319aa4e" (UID: "12722cfa-83e8-4c10-a373-81997319aa4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:45:43.315720 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315631 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "12722cfa-83e8-4c10-a373-81997319aa4e" (UID: "12722cfa-83e8-4c10-a373-81997319aa4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:45:43.315720 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315709 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-home" (OuterVolumeSpecName: "home") pod "12722cfa-83e8-4c10-a373-81997319aa4e" (UID: "12722cfa-83e8-4c10-a373-81997319aa4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:45:43.315825 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315734 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:43.315825 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.315747 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:43.317639 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.317616 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-dshm" (OuterVolumeSpecName: "dshm") pod "12722cfa-83e8-4c10-a373-81997319aa4e" (UID: "12722cfa-83e8-4c10-a373-81997319aa4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:45:43.317712 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.317644 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12722cfa-83e8-4c10-a373-81997319aa4e-kube-api-access-4hb8x" (OuterVolumeSpecName: "kube-api-access-4hb8x") pod "12722cfa-83e8-4c10-a373-81997319aa4e" (UID: "12722cfa-83e8-4c10-a373-81997319aa4e"). InnerVolumeSpecName "kube-api-access-4hb8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:45:43.317712 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.317699 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12722cfa-83e8-4c10-a373-81997319aa4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12722cfa-83e8-4c10-a373-81997319aa4e" (UID: "12722cfa-83e8-4c10-a373-81997319aa4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:45:43.416466 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.416430 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:43.416466 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.416461 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12722cfa-83e8-4c10-a373-81997319aa4e-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:43.416466 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.416473 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hb8x\" (UniqueName: \"kubernetes.io/projected/12722cfa-83e8-4c10-a373-81997319aa4e-kube-api-access-4hb8x\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:43.416692 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.416484 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12722cfa-83e8-4c10-a373-81997319aa4e-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:43.441539 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:43.441500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:44.048286 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:44.043813 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" event={"ID":"12722cfa-83e8-4c10-a373-81997319aa4e","Type":"ContainerDied","Data":"0a79c19f6b6b5aaff70273ed53d24db968f83c1310dbbb3bdda033800b5405cc"} Sep 16 19:45:44.048286 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:44.043945 2562 scope.go:117] "RemoveContainer" containerID="48f485cba2f67d4c20c4592d99473ccbae9c67c864b7e3a364c247d336beba7b" Sep 16 19:45:44.048286 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:44.044206 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz" Sep 16 19:45:44.065635 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:44.065606 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz"] Sep 16 19:45:44.073437 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:44.073408 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-dxrfz"] Sep 16 19:45:45.262405 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:45.262372 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:45.761704 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:45.761670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:45:45.961770 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:45.961739 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12722cfa-83e8-4c10-a373-81997319aa4e" path="/var/lib/kubelet/pods/12722cfa-83e8-4c10-a373-81997319aa4e/volumes" Sep 16 19:45:47.020202 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.020151 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" podUID="d691a222-56c1-4d4c-8eec-7480e3d154ec" containerName="main" containerID="cri-o://bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d" gracePeriod=45 Sep 16 19:45:47.269110 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.269077 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:45:47.345060 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.345001 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-dshm\") pod \"d691a222-56c1-4d4c-8eec-7480e3d154ec\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " Sep 16 19:45:47.345241 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.345090 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-tmp-dir\") pod \"d691a222-56c1-4d4c-8eec-7480e3d154ec\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " Sep 16 19:45:47.345241 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.345116 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-home\") pod \"d691a222-56c1-4d4c-8eec-7480e3d154ec\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " Sep 16 19:45:47.345241 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.345135 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-model-cache\") pod \"d691a222-56c1-4d4c-8eec-7480e3d154ec\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " Sep 16 19:45:47.345241 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.345156 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6dkb\" (UniqueName: \"kubernetes.io/projected/d691a222-56c1-4d4c-8eec-7480e3d154ec-kube-api-access-q6dkb\") pod \"d691a222-56c1-4d4c-8eec-7480e3d154ec\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " Sep 16 19:45:47.345241 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.345180 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d691a222-56c1-4d4c-8eec-7480e3d154ec-tls-certs\") pod \"d691a222-56c1-4d4c-8eec-7480e3d154ec\" (UID: \"d691a222-56c1-4d4c-8eec-7480e3d154ec\") " Sep 16 19:45:47.345507 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.345397 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d691a222-56c1-4d4c-8eec-7480e3d154ec" (UID: "d691a222-56c1-4d4c-8eec-7480e3d154ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:45:47.345507 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.345426 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "d691a222-56c1-4d4c-8eec-7480e3d154ec" (UID: "d691a222-56c1-4d4c-8eec-7480e3d154ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:45:47.345507 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.345404 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-home" (OuterVolumeSpecName: "home") pod "d691a222-56c1-4d4c-8eec-7480e3d154ec" (UID: "d691a222-56c1-4d4c-8eec-7480e3d154ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:45:47.347256 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.347228 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d691a222-56c1-4d4c-8eec-7480e3d154ec-kube-api-access-q6dkb" (OuterVolumeSpecName: "kube-api-access-q6dkb") pod "d691a222-56c1-4d4c-8eec-7480e3d154ec" (UID: "d691a222-56c1-4d4c-8eec-7480e3d154ec"). InnerVolumeSpecName "kube-api-access-q6dkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:45:47.347357 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.347273 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-dshm" (OuterVolumeSpecName: "dshm") pod "d691a222-56c1-4d4c-8eec-7480e3d154ec" (UID: "d691a222-56c1-4d4c-8eec-7480e3d154ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:45:47.347357 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.347288 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d691a222-56c1-4d4c-8eec-7480e3d154ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d691a222-56c1-4d4c-8eec-7480e3d154ec" (UID: "d691a222-56c1-4d4c-8eec-7480e3d154ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:45:47.445731 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.445688 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d691a222-56c1-4d4c-8eec-7480e3d154ec-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:47.445731 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.445726 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:47.445939 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.445756 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:47.445939 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.445764 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:47.445939 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.445772 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d691a222-56c1-4d4c-8eec-7480e3d154ec-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:47.445939 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:47.445780 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6dkb\" (UniqueName: \"kubernetes.io/projected/d691a222-56c1-4d4c-8eec-7480e3d154ec-kube-api-access-q6dkb\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:45:48.059911 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:48.059876 2562 generic.go:356] "Generic (PLEG): container finished" podID="d691a222-56c1-4d4c-8eec-7480e3d154ec" containerID="bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d" exitCode=0 Sep 16 19:45:48.060335 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:48.059921 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" Sep 16 19:45:48.060335 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:48.059967 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" event={"ID":"d691a222-56c1-4d4c-8eec-7480e3d154ec","Type":"ContainerDied","Data":"bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d"} Sep 16 19:45:48.060335 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:48.060047 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz" event={"ID":"d691a222-56c1-4d4c-8eec-7480e3d154ec","Type":"ContainerDied","Data":"4c00222087cfe48cef0a9c11e163efedf90006c69d8feeeff08122b0f6106454"} Sep 16 19:45:48.060335 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:48.060074 2562 scope.go:117] "RemoveContainer" containerID="bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d" Sep 16 19:45:48.067925 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:48.067896 2562 scope.go:117] "RemoveContainer" containerID="bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d" Sep 16 19:45:48.068206 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:45:48.068187 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d\": container with ID starting with bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d not found: ID does not exist" containerID="bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d" Sep 16 19:45:48.068258 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:48.068215 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d"} err="failed to get container status \"bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d\": rpc error: code = NotFound desc = could not find container \"bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d\": container with ID starting with bff3db97e742f47e249ff230d22c236035e1e92e7400ffb9fdea5e8a57f1555d not found: ID does not exist" Sep 16 19:45:48.076800 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:48.076776 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz"] Sep 16 19:45:48.082554 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:48.082532 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wbfsz"] Sep 16 19:45:49.967156 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:49.967116 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d691a222-56c1-4d4c-8eec-7480e3d154ec" path="/var/lib/kubelet/pods/d691a222-56c1-4d4c-8eec-7480e3d154ec/volumes" Sep 16 19:45:54.974474 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:45:54.974438 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:46:06.430199 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:46:06.430163 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:46:09.597478 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:46:09.597444 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:46:15.688477 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:46:15.688439 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:46:15.765940 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:46:15.765906 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:46:17.954094 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:46:17.954060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:46:18.049824 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:46:18.049780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:46:39.434828 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:46:39.434794 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:06.962210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:06.962170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:07.062540 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:07.062506 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:13.649405 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:13.649372 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:19.585426 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:19.585390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:19.672695 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:19.672659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:24.683357 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:24.683330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:26.023776 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:26.023736 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:27.900309 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:27.900270 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:28.014963 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:28.014930 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:28.108674 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:28.108643 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:32.979239 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:32.979208 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:37.786780 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:37.786746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:39.362355 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:39.362319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:40.181693 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:40.181634 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:49.395529 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:49.395492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:53.313704 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:53.313666 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:47:55.853375 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:47:55.853339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:10.475686 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:10.475645 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:16.963913 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:16.963833 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:17.136238 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:17.136207 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:21.967987 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:21.967956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:22.211152 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:22.211117 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:26.348869 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:26.348837 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:32.099993 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:32.099952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:42.985273 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:42.985228 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:43.086342 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:43.086308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:43.177032 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:43.176985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:48:48.035274 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:48:48.035242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:49:07.260025 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.259981 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc"] Sep 16 19:49:07.260407 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.260245 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12722cfa-83e8-4c10-a373-81997319aa4e" containerName="main" Sep 16 19:49:07.260407 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.260256 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="12722cfa-83e8-4c10-a373-81997319aa4e" containerName="main" Sep 16 19:49:07.260407 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.260275 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d691a222-56c1-4d4c-8eec-7480e3d154ec" containerName="main" Sep 16 19:49:07.260407 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.260281 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d691a222-56c1-4d4c-8eec-7480e3d154ec" containerName="main" Sep 16 19:49:07.260407 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.260326 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d691a222-56c1-4d4c-8eec-7480e3d154ec" containerName="main" Sep 16 19:49:07.260407 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.260336 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="12722cfa-83e8-4c10-a373-81997319aa4e" containerName="main" Sep 16 19:49:07.263140 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.263124 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.266887 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.266861 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\"" Sep 16 19:49:07.267901 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.267881 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-d74ps\"" Sep 16 19:49:07.268042 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.267939 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\"" Sep 16 19:49:07.268130 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.268090 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\"" Sep 16 19:49:07.289090 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.289061 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc"] Sep 16 19:49:07.333220 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.333190 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:49:07.436852 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.436815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-home\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.437069 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.436879 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.437069 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.436903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a221c56-8572-4b49-b3cb-873c5eb2b330-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.437069 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.436930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8r4\" (UniqueName: \"kubernetes.io/projected/3a221c56-8572-4b49-b3cb-873c5eb2b330-kube-api-access-4g8r4\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.437069 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.436961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.437069 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.437034 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-dshm\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.537998 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.537902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8r4\" (UniqueName: \"kubernetes.io/projected/3a221c56-8572-4b49-b3cb-873c5eb2b330-kube-api-access-4g8r4\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.537998 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.537948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.537998 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.537985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-dshm\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.538314 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.538007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-home\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.538314 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.538058 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.538314 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.538076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a221c56-8572-4b49-b3cb-873c5eb2b330-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.538516 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.538493 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.538584 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.538534 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.538627 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.538593 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-home\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.540229 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.540208 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-dshm\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.540475 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.540459 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a221c56-8572-4b49-b3cb-873c5eb2b330-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.547771 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.547751 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8r4\" (UniqueName: \"kubernetes.io/projected/3a221c56-8572-4b49-b3cb-873c5eb2b330-kube-api-access-4g8r4\") pod \"roll-v1-kserve-69cf8bcfc-27fsc\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.573811 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.573763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:07.850156 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:07.850122 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc"] Sep 16 19:49:07.853114 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:49:07.853087 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a221c56_8572_4b49_b3cb_873c5eb2b330.slice/crio-fc762616bb0832d430e335ad2be01ebda744bacff7cbcd94a9f9a33b94b4179d WatchSource:0}: Error finding container fc762616bb0832d430e335ad2be01ebda744bacff7cbcd94a9f9a33b94b4179d: Status 404 returned error can't find the container with id fc762616bb0832d430e335ad2be01ebda744bacff7cbcd94a9f9a33b94b4179d Sep 16 19:49:08.793437 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:08.789878 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" event={"ID":"3a221c56-8572-4b49-b3cb-873c5eb2b330","Type":"ContainerStarted","Data":"3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9"} Sep 16 19:49:08.793437 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:08.789986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" event={"ID":"3a221c56-8572-4b49-b3cb-873c5eb2b330","Type":"ContainerStarted","Data":"fc762616bb0832d430e335ad2be01ebda744bacff7cbcd94a9f9a33b94b4179d"} Sep 16 19:49:08.824003 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:08.823954 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" podStartSLOduration=1.823939652 podStartE2EDuration="1.823939652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:08.821607396 +0000 UTC m=+2657.370664670" watchObservedRunningTime="2026-09-16 19:49:08.823939652 +0000 UTC m=+2657.372996925" Sep 16 19:49:09.573971 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:09.573937 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:17.574086 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:17.574052 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:17.586373 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:17.586350 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:17.832223 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:17.832137 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:49:19.569975 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.569940 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44"] Sep 16 19:49:19.573413 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.573390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.577007 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.576985 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\"" Sep 16 19:49:19.587263 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.587240 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44"] Sep 16 19:49:19.636755 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.636712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.636755 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.636755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670cae35-f55a-43de-a349-cfba23a45cd0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.636957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.636810 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lhtm\" (UniqueName: \"kubernetes.io/projected/670cae35-f55a-43de-a349-cfba23a45cd0-kube-api-access-8lhtm\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.636957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.636844 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.636957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.636923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-home\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.637123 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.637056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.726734 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.726703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:49:19.737959 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.737926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.738116 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.737964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.738116 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.737983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670cae35-f55a-43de-a349-cfba23a45cd0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.738116 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.738001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lhtm\" (UniqueName: \"kubernetes.io/projected/670cae35-f55a-43de-a349-cfba23a45cd0-kube-api-access-8lhtm\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.738116 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.738038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.738116 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.738077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-home\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.738457 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.738430 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.738522 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.738430 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-home\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.738522 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.738493 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.740497 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.740477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.740616 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.740597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670cae35-f55a-43de-a349-cfba23a45cd0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.751496 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.751477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lhtm\" (UniqueName: \"kubernetes.io/projected/670cae35-f55a-43de-a349-cfba23a45cd0-kube-api-access-8lhtm\") pod \"roll-v2-kserve-d4dbf9d67-7fv44\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:19.883779 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:19.883741 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:20.160453 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:20.160375 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44"] Sep 16 19:49:20.163241 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:49:20.163209 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670cae35_f55a_43de_a349_cfba23a45cd0.slice/crio-0c601b34a558e86cd6b04dc78fe28b9f9dad5bfe3e2f5381e60af146e275bd43 WatchSource:0}: Error finding container 0c601b34a558e86cd6b04dc78fe28b9f9dad5bfe3e2f5381e60af146e275bd43: Status 404 returned error can't find the container with id 0c601b34a558e86cd6b04dc78fe28b9f9dad5bfe3e2f5381e60af146e275bd43 Sep 16 19:49:20.840443 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:20.840403 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" event={"ID":"670cae35-f55a-43de-a349-cfba23a45cd0","Type":"ContainerStarted","Data":"0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0"} Sep 16 19:49:20.840817 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:20.840451 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" event={"ID":"670cae35-f55a-43de-a349-cfba23a45cd0","Type":"ContainerStarted","Data":"0c601b34a558e86cd6b04dc78fe28b9f9dad5bfe3e2f5381e60af146e275bd43"} Sep 16 19:49:20.865184 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:20.865137 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" podStartSLOduration=1.865120048 podStartE2EDuration="1.865120048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:20.862788363 +0000 UTC m=+2669.411845636" watchObservedRunningTime="2026-09-16 19:49:20.865120048 +0000 UTC m=+2669.414177323" Sep 16 19:49:20.884435 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:20.884406 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:26.133854 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:26.133816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:49:29.884192 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:29.884160 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:29.896495 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:29.896474 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:30.889521 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:30.889489 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:49:32.014575 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:32.014531 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:49:32.098004 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:32.097970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:49:33.693201 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:33.693167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:49:33.775635 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:33.775598 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:49:52.049476 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:52.049397 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:49:52.054550 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:49:52.054524 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:50:22.636421 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:22.636387 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:22.733579 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:22.733549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:36.377847 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:36.377814 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:36.544101 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:36.544070 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:36.783237 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:36.783134 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:37.069868 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:37.069782 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc"] Sep 16 19:50:37.104269 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:37.104231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:37.652295 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:37.652266 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44"] Sep 16 19:50:37.670384 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:37.670355 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:41.453181 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:41.453146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:50.211322 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:50.211287 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:50.290671 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:50.290604 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:50:52.088658 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.088600 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" podUID="3a221c56-8572-4b49-b3cb-873c5eb2b330" containerName="main" containerID="cri-o://3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9" gracePeriod=45 Sep 16 19:50:52.327615 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.327592 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:50:52.393861 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.393777 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a221c56-8572-4b49-b3cb-873c5eb2b330-tls-certs\") pod \"3a221c56-8572-4b49-b3cb-873c5eb2b330\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " Sep 16 19:50:52.393861 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.393815 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g8r4\" (UniqueName: \"kubernetes.io/projected/3a221c56-8572-4b49-b3cb-873c5eb2b330-kube-api-access-4g8r4\") pod \"3a221c56-8572-4b49-b3cb-873c5eb2b330\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " Sep 16 19:50:52.394073 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.393887 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-tmp-dir\") pod \"3a221c56-8572-4b49-b3cb-873c5eb2b330\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " Sep 16 19:50:52.394073 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.393917 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-model-cache\") pod \"3a221c56-8572-4b49-b3cb-873c5eb2b330\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " Sep 16 19:50:52.394073 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.393946 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-dshm\") pod \"3a221c56-8572-4b49-b3cb-873c5eb2b330\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " Sep 16 19:50:52.394073 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.393985 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-home\") pod \"3a221c56-8572-4b49-b3cb-873c5eb2b330\" (UID: \"3a221c56-8572-4b49-b3cb-873c5eb2b330\") " Sep 16 19:50:52.394286 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.394174 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a221c56-8572-4b49-b3cb-873c5eb2b330" (UID: "3a221c56-8572-4b49-b3cb-873c5eb2b330"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:52.394286 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.394189 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a221c56-8572-4b49-b3cb-873c5eb2b330" (UID: "3a221c56-8572-4b49-b3cb-873c5eb2b330"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:52.394392 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.394267 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:52.394392 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.394306 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:52.394392 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.394334 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-home" (OuterVolumeSpecName: "home") pod "3a221c56-8572-4b49-b3cb-873c5eb2b330" (UID: "3a221c56-8572-4b49-b3cb-873c5eb2b330"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:52.396077 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.396055 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-dshm" (OuterVolumeSpecName: "dshm") pod "3a221c56-8572-4b49-b3cb-873c5eb2b330" (UID: "3a221c56-8572-4b49-b3cb-873c5eb2b330"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:52.396418 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.396390 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a221c56-8572-4b49-b3cb-873c5eb2b330-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a221c56-8572-4b49-b3cb-873c5eb2b330" (UID: "3a221c56-8572-4b49-b3cb-873c5eb2b330"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:50:52.396418 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.396407 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a221c56-8572-4b49-b3cb-873c5eb2b330-kube-api-access-4g8r4" (OuterVolumeSpecName: "kube-api-access-4g8r4") pod "3a221c56-8572-4b49-b3cb-873c5eb2b330" (UID: "3a221c56-8572-4b49-b3cb-873c5eb2b330"). InnerVolumeSpecName "kube-api-access-4g8r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:50:52.495090 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.495061 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a221c56-8572-4b49-b3cb-873c5eb2b330-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:52.495090 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.495082 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8r4\" (UniqueName: \"kubernetes.io/projected/3a221c56-8572-4b49-b3cb-873c5eb2b330-kube-api-access-4g8r4\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:52.495090 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.495091 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:52.495302 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.495100 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a221c56-8572-4b49-b3cb-873c5eb2b330-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:52.670496 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.670410 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" podUID="670cae35-f55a-43de-a349-cfba23a45cd0" containerName="main" containerID="cri-o://0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0" gracePeriod=45 Sep 16 19:50:52.899418 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.899388 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:50:52.999717 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.999617 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-tmp-dir\") pod \"670cae35-f55a-43de-a349-cfba23a45cd0\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " Sep 16 19:50:52.999717 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.999668 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lhtm\" (UniqueName: \"kubernetes.io/projected/670cae35-f55a-43de-a349-cfba23a45cd0-kube-api-access-8lhtm\") pod \"670cae35-f55a-43de-a349-cfba23a45cd0\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " Sep 16 19:50:52.999954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.999726 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670cae35-f55a-43de-a349-cfba23a45cd0-tls-certs\") pod \"670cae35-f55a-43de-a349-cfba23a45cd0\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " Sep 16 19:50:52.999954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.999743 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-dshm\") pod \"670cae35-f55a-43de-a349-cfba23a45cd0\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " Sep 16 19:50:52.999954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.999763 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-home\") pod \"670cae35-f55a-43de-a349-cfba23a45cd0\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " Sep 16 19:50:52.999954 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.999802 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-model-cache\") pod \"670cae35-f55a-43de-a349-cfba23a45cd0\" (UID: \"670cae35-f55a-43de-a349-cfba23a45cd0\") " Sep 16 19:50:53.000149 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:52.999930 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "670cae35-f55a-43de-a349-cfba23a45cd0" (UID: "670cae35-f55a-43de-a349-cfba23a45cd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:53.000149 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.000060 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-home" (OuterVolumeSpecName: "home") pod "670cae35-f55a-43de-a349-cfba23a45cd0" (UID: "670cae35-f55a-43de-a349-cfba23a45cd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:53.000149 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.000077 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-model-cache" (OuterVolumeSpecName: "model-cache") pod "670cae35-f55a-43de-a349-cfba23a45cd0" (UID: "670cae35-f55a-43de-a349-cfba23a45cd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:53.001905 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.001883 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-dshm" (OuterVolumeSpecName: "dshm") pod "670cae35-f55a-43de-a349-cfba23a45cd0" (UID: "670cae35-f55a-43de-a349-cfba23a45cd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:53.001971 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.001921 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670cae35-f55a-43de-a349-cfba23a45cd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "670cae35-f55a-43de-a349-cfba23a45cd0" (UID: "670cae35-f55a-43de-a349-cfba23a45cd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:50:53.001971 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.001938 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670cae35-f55a-43de-a349-cfba23a45cd0-kube-api-access-8lhtm" (OuterVolumeSpecName: "kube-api-access-8lhtm") pod "670cae35-f55a-43de-a349-cfba23a45cd0" (UID: "670cae35-f55a-43de-a349-cfba23a45cd0"). InnerVolumeSpecName "kube-api-access-8lhtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:50:53.101114 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.101073 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:53.101114 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.101114 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:53.101445 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.101126 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:53.101445 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.101140 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670cae35-f55a-43de-a349-cfba23a45cd0-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:53.101445 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.101152 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lhtm\" (UniqueName: \"kubernetes.io/projected/670cae35-f55a-43de-a349-cfba23a45cd0-kube-api-access-8lhtm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:53.101445 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.101164 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670cae35-f55a-43de-a349-cfba23a45cd0-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:50:53.191639 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.191600 2562 generic.go:356] "Generic (PLEG): container finished" podID="670cae35-f55a-43de-a349-cfba23a45cd0" containerID="0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0" exitCode=0 Sep 16 19:50:53.191840 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.191648 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" Sep 16 19:50:53.191840 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.191692 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" event={"ID":"670cae35-f55a-43de-a349-cfba23a45cd0","Type":"ContainerDied","Data":"0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0"} Sep 16 19:50:53.191840 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.191747 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" event={"ID":"670cae35-f55a-43de-a349-cfba23a45cd0","Type":"ContainerDied","Data":"0c601b34a558e86cd6b04dc78fe28b9f9dad5bfe3e2f5381e60af146e275bd43"} Sep 16 19:50:53.191840 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.191781 2562 scope.go:117] "RemoveContainer" containerID="0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0" Sep 16 19:50:53.193218 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.193048 2562 generic.go:356] "Generic (PLEG): container finished" podID="3a221c56-8572-4b49-b3cb-873c5eb2b330" containerID="3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9" exitCode=0 Sep 16 19:50:53.193218 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.193102 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" Sep 16 19:50:53.193218 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.193113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" event={"ID":"3a221c56-8572-4b49-b3cb-873c5eb2b330","Type":"ContainerDied","Data":"3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9"} Sep 16 19:50:53.193218 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.193161 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc" event={"ID":"3a221c56-8572-4b49-b3cb-873c5eb2b330","Type":"ContainerDied","Data":"fc762616bb0832d430e335ad2be01ebda744bacff7cbcd94a9f9a33b94b4179d"} Sep 16 19:50:53.199857 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.199838 2562 scope.go:117] "RemoveContainer" containerID="0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0" Sep 16 19:50:53.200142 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:50:53.200123 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0\": container with ID starting with 0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0 not found: ID does not exist" containerID="0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0" Sep 16 19:50:53.200197 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.200151 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0"} err="failed to get container status \"0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0\": rpc error: code = NotFound desc = could not find container \"0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0\": container with ID starting with 0f1d59d2219ba6d0a109cf0ea817ccc436dd01fcaa1d9047115be868db5fd3e0 not found: ID does not exist" Sep 16 19:50:53.200197 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.200167 2562 scope.go:117] "RemoveContainer" containerID="3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9" Sep 16 19:50:53.209726 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.209703 2562 scope.go:117] "RemoveContainer" containerID="3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9" Sep 16 19:50:53.210203 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:50:53.210183 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9\": container with ID starting with 3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9 not found: ID does not exist" containerID="3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9" Sep 16 19:50:53.210265 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.210211 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9"} err="failed to get container status \"3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9\": rpc error: code = NotFound desc = could not find container \"3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9\": container with ID starting with 3ea007ccba674cfa1cbdb0fa147502c1ba1b7049a8ca7aa32f88ebed131818c9 not found: ID does not exist" Sep 16 19:50:53.220744 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.217755 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44"] Sep 16 19:50:53.222490 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.222470 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44"] Sep 16 19:50:53.233908 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.233886 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc"] Sep 16 19:50:53.236210 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.236191 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-27fsc"] Sep 16 19:50:53.879973 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.879918 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7fv44" podUID="670cae35-f55a-43de-a349-cfba23a45cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": context deadline exceeded" Sep 16 19:50:53.962126 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.962096 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a221c56-8572-4b49-b3cb-873c5eb2b330" path="/var/lib/kubelet/pods/3a221c56-8572-4b49-b3cb-873c5eb2b330/volumes" Sep 16 19:50:53.962485 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:50:53.962472 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="670cae35-f55a-43de-a349-cfba23a45cd0" path="/var/lib/kubelet/pods/670cae35-f55a-43de-a349-cfba23a45cd0/volumes" Sep 16 19:51:02.672989 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:02.672928 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:04.022452 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:04.022420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:04.128235 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:04.128198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:08.465995 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:08.465962 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:11.849199 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:11.849170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:11.949380 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:11.949344 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:19.315647 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:19.315570 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:19.493416 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:19.493382 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:25.110607 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:25.110554 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:25.210745 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:25.210713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:25.395340 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:25.395224 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:25.465358 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:25.465326 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:39.767417 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:39.767376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:51:40.097201 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:51:40.097161 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:52:02.095000 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:52:02.094946 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:52:03.876719 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:52:03.876686 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:52:20.342373 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:52:20.342337 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:52:25.846592 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:52:25.846559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:52:26.379810 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:52:26.379764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:52:27.778442 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:52:27.778411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:52:30.055256 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:52:30.055219 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:52:31.450532 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:52:31.450503 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:08.753628 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:08.753593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:08.887192 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:08.887157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:09.000093 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:09.000064 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:09.317232 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:09.317196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:15.040440 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:15.040405 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:15.094944 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:15.094912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:17.549254 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.549215 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794"] Sep 16 19:53:17.549606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.549505 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a221c56-8572-4b49-b3cb-873c5eb2b330" containerName="main" Sep 16 19:53:17.549606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.549517 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a221c56-8572-4b49-b3cb-873c5eb2b330" containerName="main" Sep 16 19:53:17.549606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.549543 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670cae35-f55a-43de-a349-cfba23a45cd0" containerName="main" Sep 16 19:53:17.549606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.549550 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="670cae35-f55a-43de-a349-cfba23a45cd0" containerName="main" Sep 16 19:53:17.549606 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.549606 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a221c56-8572-4b49-b3cb-873c5eb2b330" containerName="main" Sep 16 19:53:17.549775 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.549614 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="670cae35-f55a-43de-a349-cfba23a45cd0" containerName="main" Sep 16 19:53:17.552304 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.552287 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.555237 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.555218 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\"" Sep 16 19:53:17.556242 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.556224 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-lmmlj\"" Sep 16 19:53:17.556348 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.556254 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\"" Sep 16 19:53:17.556348 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.556311 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\"" Sep 16 19:53:17.564323 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.564302 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794"] Sep 16 19:53:17.579423 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.579391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:17.596524 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.596495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.596656 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.596534 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.596656 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.596554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rwh\" (UniqueName: \"kubernetes.io/projected/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-kube-api-access-n9rwh\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.596656 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.596615 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.596656 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.596645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.596795 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.596690 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.697692 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.697654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.697881 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.697704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.697881 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.697731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rwh\" (UniqueName: \"kubernetes.io/projected/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-kube-api-access-n9rwh\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.697881 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.697758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.698083 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.697868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.698083 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.697929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.698198 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.698154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.698257 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.698217 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.698355 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.698335 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.700032 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.699995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.700344 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.700329 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.710079 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.710056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rwh\" (UniqueName: \"kubernetes.io/projected/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-kube-api-access-n9rwh\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:17.862197 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:17.862158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:18.134249 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:18.134225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794"] Sep 16 19:53:18.136228 ip-10-0-139-17 kubenswrapper[2562]: W0916 19:53:18.136204 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aed0058_efec_4a0c_9ebb_24fa6ed3e8a7.slice/crio-c2c6f0141cf042e8d9ef396cafe989cba945454a6987f8ce6362a877982bfc81 WatchSource:0}: Error finding container c2c6f0141cf042e8d9ef396cafe989cba945454a6987f8ce6362a877982bfc81: Status 404 returned error can't find the container with id c2c6f0141cf042e8d9ef396cafe989cba945454a6987f8ce6362a877982bfc81 Sep 16 19:53:18.730029 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:18.729968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" event={"ID":"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7","Type":"ContainerStarted","Data":"33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a"} Sep 16 19:53:18.730422 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:18.730047 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" event={"ID":"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7","Type":"ContainerStarted","Data":"c2c6f0141cf042e8d9ef396cafe989cba945454a6987f8ce6362a877982bfc81"} Sep 16 19:53:18.863352 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:18.863315 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:27.546732 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:27.546695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:27.862668 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:27.862630 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:27.874813 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:27.874781 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:27.893430 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:27.893384 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" podStartSLOduration=10.893367925 podStartE2EDuration="10.893367925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:53:18.748129326 +0000 UTC m=+2907.297186601" watchObservedRunningTime="2026-09-16 19:53:27.893367925 +0000 UTC m=+2916.442425199" Sep 16 19:53:28.777739 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:28.777708 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:43.083001 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.082968 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794"] Sep 16 19:53:43.091524 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:53:43.091483 2562 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:53:43.091524 ip-10-0-139-17 kubenswrapper[2562]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:53:43.091524 ip-10-0-139-17 kubenswrapper[2562]: , stdout: , stderr: , exit code -1 Sep 16 19:53:43.091524 ip-10-0-139-17 kubenswrapper[2562]: > containerID="33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a" cmd=["/bin/sleep","15"] Sep 16 19:53:43.091808 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.091538 2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" message="" err=< Sep 16 19:53:43.091808 ip-10-0-139-17 kubenswrapper[2562]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:53:43.091808 ip-10-0-139-17 kubenswrapper[2562]: , stdout: , stderr: , exit code -1 Sep 16 19:53:43.091808 ip-10-0-139-17 kubenswrapper[2562]: > Sep 16 19:53:43.091808 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:53:43.091556 2562 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:53:43.091808 ip-10-0-139-17 kubenswrapper[2562]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:53:43.091808 ip-10-0-139-17 kubenswrapper[2562]: , stdout: , stderr: , exit code -1 Sep 16 19:53:43.091808 ip-10-0-139-17 kubenswrapper[2562]: > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" podUID="5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" containerName="main" containerID="cri-o://33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a" Sep 16 19:53:43.091808 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.091600 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" podUID="5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" containerName="main" containerID="cri-o://33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a" gracePeriod=60 Sep 16 19:53:43.102526 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.102496 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:43.431748 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.431727 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:43.505759 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.505721 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-dshm\") pod \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " Sep 16 19:53:43.505929 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.505774 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tls-certs\") pod \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " Sep 16 19:53:43.505929 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.505816 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9rwh\" (UniqueName: \"kubernetes.io/projected/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-kube-api-access-n9rwh\") pod \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " Sep 16 19:53:43.505929 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.505839 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-model-cache\") pod \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " Sep 16 19:53:43.505929 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.505872 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-home\") pod \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " Sep 16 19:53:43.505929 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.505901 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tmp-dir\") pod \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\" (UID: \"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7\") " Sep 16 19:53:43.506208 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.506132 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" (UID: "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:53:43.506208 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.506143 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-home" (OuterVolumeSpecName: "home") pod "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" (UID: "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:53:43.506314 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.506259 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" (UID: "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:53:43.508180 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.508157 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-kube-api-access-n9rwh" (OuterVolumeSpecName: "kube-api-access-n9rwh") pod "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" (UID: "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7"). InnerVolumeSpecName "kube-api-access-n9rwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:53:43.508180 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.508158 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-dshm" (OuterVolumeSpecName: "dshm") pod "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" (UID: "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:53:43.508371 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.508158 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" (UID: "5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:53:43.606635 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.606556 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9rwh\" (UniqueName: \"kubernetes.io/projected/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-kube-api-access-n9rwh\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:53:43.606635 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.606580 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:53:43.606635 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.606590 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:53:43.606635 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.606600 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:53:43.606635 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.606607 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:53:43.606635 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.606614 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 19:53:43.823665 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.823625 2562 generic.go:356] "Generic (PLEG): container finished" podID="5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" containerID="33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a" exitCode=0 Sep 16 19:53:43.823665 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.823671 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" Sep 16 19:53:43.823889 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.823699 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" event={"ID":"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7","Type":"ContainerDied","Data":"33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a"} Sep 16 19:53:43.823889 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.823752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794" event={"ID":"5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7","Type":"ContainerDied","Data":"c2c6f0141cf042e8d9ef396cafe989cba945454a6987f8ce6362a877982bfc81"} Sep 16 19:53:43.823889 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.823775 2562 scope.go:117] "RemoveContainer" containerID="33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a" Sep 16 19:53:43.832787 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.832768 2562 scope.go:117] "RemoveContainer" containerID="33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a" Sep 16 19:53:43.833100 ip-10-0-139-17 kubenswrapper[2562]: E0916 19:53:43.833080 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a\": container with ID starting with 33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a not found: ID does not exist" containerID="33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a" Sep 16 19:53:43.833182 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.833111 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a"} err="failed to get container status \"33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a\": rpc error: code = NotFound desc = could not find container \"33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a\": container with ID starting with 33ff12155b9ef5788e981cd14391925433d9f6174603bbc983f7415a4fdab27a not found: ID does not exist" Sep 16 19:53:43.844549 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.844520 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794"] Sep 16 19:53:43.846704 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.846682 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dx794"] Sep 16 19:53:43.962470 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:43.962395 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" path="/var/lib/kubelet/pods/5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7/volumes" Sep 16 19:53:46.018588 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:46.018554 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:46.098349 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:46.098313 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:47.715517 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:47.715485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:47.819157 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:47.819121 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:53:52.746686 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:53:52.746649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:03.478957 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:03.478924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:03.841100 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:03.841067 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:14.054284 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:14.054242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:15.363258 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:15.363220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:21.006609 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:21.006561 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:30.425444 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:30.425398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:30.524661 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:30.524628 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:34.100065 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:34.100028 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:34.177812 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:34.177783 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:36.586680 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:36.586646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:36.659739 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:36.659702 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:54:52.069988 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:52.069960 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:54:52.077967 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:54:52.077942 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:55:24.166792 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:55:24.166760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:55:24.261200 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:55:24.261165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:55:29.110518 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:55:29.110489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:55:29.336976 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:55:29.336915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:55:31.949979 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:55:31.949948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:55:37.604352 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:55:37.604311 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:55:37.667212 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:55:37.667090 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:55:44.326132 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:55:44.326021 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:56:09.120684 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:56:09.120650 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:56:09.222244 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:56:09.222207 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:56:59.065570 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:56:59.065534 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:56:59.161051 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:56:59.160994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:03.821584 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:03.821547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:03.921329 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:03.921297 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:05.765497 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:05.765463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:11.421655 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:11.421619 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:11.521637 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:11.521597 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:23.604915 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:23.604833 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:40.713471 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:40.713438 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:41.988146 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:41.988110 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:46.278925 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:46.278884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:47.984439 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:47.984405 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:48.085610 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:48.085578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:56.045765 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:56.045726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:57:56.144705 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:57:56.144662 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:09.393151 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:09.393113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:09.480303 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:09.480278 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:33.736369 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:33.736332 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:33.852546 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:33.852510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:33.987217 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:33.987114 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:40.142892 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:40.142864 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:45.871763 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:45.871642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:45.937287 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:45.937255 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:53.606189 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:53.606157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:58:53.823952 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:58:53.823915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:59:11.204523 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:59:11.204490 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:59:52.092644 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:59:52.092609 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:59:52.100046 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:59:52.100022 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 19:59:56.116716 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:59:56.116675 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 19:59:56.190539 ip-10-0-139-17 kubenswrapper[2562]: I0916 19:59:56.190481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:15.743751 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:15.743673 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:15.841176 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:15.841143 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:46.255487 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:46.255447 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:46.773346 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:46.773312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:52.571313 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:52.571276 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:54.088324 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:54.088291 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:54.430329 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:54.430250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:59.326401 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:59.326368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:59.426763 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:59.426718 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:00:59.650556 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:00:59.650522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:05.174109 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:05.174079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:05.277103 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:05.277070 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:05.376434 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:05.376401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:07.485308 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:07.485273 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:07.634565 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:07.634531 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:07.768212 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:07.768131 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:12.041097 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:12.041060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:13.463658 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:13.463619 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:13.563374 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:13.563340 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:16.078208 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:16.078168 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:16.273757 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:16.273724 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:18.375030 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:18.374977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:18.475662 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:18.475628 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:18.657923 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:18.657841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:18.754468 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:18.754436 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:19.985540 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:19.985511 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:20.087154 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:20.087124 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:20.189081 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:20.189051 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:24.290133 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:24.290099 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:24.390235 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:24.390204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:24.490899 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:24.490866 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:25.049918 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:25.049887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:28.838268 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:28.838234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:29.038580 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:29.038537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:29.136958 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:29.136876 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:31.279646 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:31.279617 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:31.381295 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:31.381262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:34.891224 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:34.891187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:35.260419 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:35.260208 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:35.357700 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:35.357624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:40.090309 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:40.090273 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:40.367308 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:40.367225 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:42.178105 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:42.178071 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:42.241828 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:42.241795 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:43.577966 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:43.577942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:43.677707 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:43.677674 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:48.654589 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:48.654513 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:49.189847 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:49.189811 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:01:49.467796 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:01:49.467719 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:02:01.198425 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:02:01.198387 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:02:24.273071 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:02:24.273040 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:02:30.143491 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:02:30.143452 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:02:55.797752 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:02:55.797716 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:02:58.934836 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:02:58.934797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:02:59.102491 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:02:59.102441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:03:04.164742 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:03:04.164716 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:03:49.039415 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:03:49.039379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:03:49.135948 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:03:49.135916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:03:49.284838 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:03:49.284804 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:03:49.359187 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:03:49.359150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:03:52.948504 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:03:52.948468 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:03:56.115821 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:03:56.115785 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:03:56.506954 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:03:56.506869 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:02.100700 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:02.100666 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:02.174659 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:02.174622 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:05.998993 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:05.998959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:06.335862 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:06.335826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:06.444234 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:06.444195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:08.949509 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:08.949471 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:09.049546 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:09.049502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:09.105350 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:09.105318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:31.682495 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:31.682463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:32.000323 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.000251 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm"] Sep 16 20:04:32.000534 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.000520 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" containerName="main" Sep 16 20:04:32.000577 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.000536 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" containerName="main" Sep 16 20:04:32.000615 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.000581 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aed0058-efec-4a0c-9ebb-24fa6ed3e8a7" containerName="main" Sep 16 20:04:32.003298 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.003282 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:04:32.007526 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.007504 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\"" Sep 16 20:04:32.007642 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.007578 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\"" Sep 16 20:04:32.008600 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.008571 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-7fzfn\"" Sep 16 20:04:32.008828 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.008575 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qqkdg\"" Sep 16 20:04:32.008950 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.008609 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\"" Sep 16 20:04:32.014756 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.014736 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm"] Sep 16 20:04:32.026412 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.026390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:32.059720 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.059686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5m8q\" (UniqueName: \"kubernetes.io/projected/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-kube-api-access-m5m8q\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-4c4dm\" (UID: \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:04:32.059720 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.059727 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-4c4dm\" (UID: \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:04:32.160088 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.160045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5m8q\" (UniqueName: \"kubernetes.io/projected/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-kube-api-access-m5m8q\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-4c4dm\" (UID: \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:04:32.160266 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.160095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-4c4dm\" (UID: \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:04:32.162558 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.162536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-4c4dm\" (UID: \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:04:32.169630 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.169604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5m8q\" (UniqueName: \"kubernetes.io/projected/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-kube-api-access-m5m8q\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-4c4dm\" (UID: \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:04:32.313631 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.313551 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:04:32.590064 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.590027 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm"] Sep 16 20:04:32.593586 ip-10-0-139-17 kubenswrapper[2562]: W0916 20:04:32.593554 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc1fe16_93f2_47ef_9c18_2e6b24dbae47.slice/crio-4124baec241e0aa06700c015f14aca9e3ad5b54a3683305cbe9512df8491b1b6 WatchSource:0}: Error finding container 4124baec241e0aa06700c015f14aca9e3ad5b54a3683305cbe9512df8491b1b6: Status 404 returned error can't find the container with id 4124baec241e0aa06700c015f14aca9e3ad5b54a3683305cbe9512df8491b1b6 Sep 16 20:04:32.595659 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:32.595631 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 20:04:33.224575 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:33.224539 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" event={"ID":"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47","Type":"ContainerStarted","Data":"4124baec241e0aa06700c015f14aca9e3ad5b54a3683305cbe9512df8491b1b6"} Sep 16 20:04:35.236158 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:35.236060 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" event={"ID":"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47","Type":"ContainerStarted","Data":"7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a"} Sep 16 20:04:35.236512 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:35.236239 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:04:35.253044 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:35.252867 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" podStartSLOduration=1.938296084 podStartE2EDuration="4.252850647s" totalImagesPullingTime="2.314554563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:35.252665581 +0000 UTC m=+3583.801722852" watchObservedRunningTime="2026-09-16 20:04:35.252850647 +0000 UTC m=+3583.801907921" Sep 16 20:04:50.681213 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:50.681172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:52.121273 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:52.121238 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:04:52.123988 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:52.123961 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:04:53.617194 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:53.617122 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:04:53.707481 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:04:53.707449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:05:06.242400 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:05:06.242371 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:06:49.284238 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:06:49.284202 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:06:54.084246 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:06:54.084207 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm"] Sep 16 20:06:54.104167 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:06:54.104130 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:06:54.185622 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:06:54.185579 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:06:56.242326 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:06:56.242288 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:07:06.241881 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:06.241839 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:07:09.104039 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:09.103960 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerName="main" containerID="cri-o://7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a" gracePeriod=45 Sep 16 20:07:16.242708 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:16.242672 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:07:16.243112 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:16.242780 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:07:26.242355 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:26.242316 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:07:36.241996 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:36.241956 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:07:39.350873 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.350847 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:07:39.393410 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.393378 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5m8q\" (UniqueName: \"kubernetes.io/projected/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-kube-api-access-m5m8q\") pod \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\" (UID: \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\") " Sep 16 20:07:39.393599 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.393452 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-tls-certs\") pod \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\" (UID: \"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47\") " Sep 16 20:07:39.395657 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.395624 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-kube-api-access-m5m8q" (OuterVolumeSpecName: "kube-api-access-m5m8q") pod "4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" (UID: "4fc1fe16-93f2-47ef-9c18-2e6b24dbae47"). InnerVolumeSpecName "kube-api-access-m5m8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:07:39.395777 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.395665 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" (UID: "4fc1fe16-93f2-47ef-9c18-2e6b24dbae47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:07:39.494797 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.494760 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:07:39.494797 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.494787 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5m8q\" (UniqueName: \"kubernetes.io/projected/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47-kube-api-access-m5m8q\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:07:39.934665 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.934633 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerID="7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a" exitCode=0 Sep 16 20:07:39.934847 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.934681 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" Sep 16 20:07:39.934847 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.934714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" event={"ID":"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47","Type":"ContainerDied","Data":"7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a"} Sep 16 20:07:39.934847 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.934771 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm" event={"ID":"4fc1fe16-93f2-47ef-9c18-2e6b24dbae47","Type":"ContainerDied","Data":"4124baec241e0aa06700c015f14aca9e3ad5b54a3683305cbe9512df8491b1b6"} Sep 16 20:07:39.934847 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.934793 2562 scope.go:117] "RemoveContainer" containerID="7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a" Sep 16 20:07:39.943615 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.943600 2562 scope.go:117] "RemoveContainer" containerID="7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a" Sep 16 20:07:39.943892 ip-10-0-139-17 kubenswrapper[2562]: E0916 20:07:39.943873 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a\": container with ID starting with 7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a not found: ID does not exist" containerID="7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a" Sep 16 20:07:39.943938 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.943900 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a"} err="failed to get container status \"7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a\": rpc error: code = NotFound desc = could not find container \"7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a\": container with ID starting with 7a2f5825091e5c6dd1ae0bc83184d2b87350a8d0097534f66dbbc5fe990b125a not found: ID does not exist" Sep 16 20:07:39.956465 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.956442 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm"] Sep 16 20:07:39.961926 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:39.961905 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-4c4dm"] Sep 16 20:07:41.962231 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:41.962199 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" path="/var/lib/kubelet/pods/4fc1fe16-93f2-47ef-9c18-2e6b24dbae47/volumes" Sep 16 20:07:43.192356 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:43.192314 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:07:43.288707 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:43.288665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:07:43.388530 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:43.388494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:07:47.403752 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:47.403659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:07:50.242198 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:50.242162 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:07:55.891114 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:55.891073 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:07:55.960921 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:07:55.960875 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:08:16.403030 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:08:16.402985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:08:35.328969 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:08:35.328934 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:08:35.401402 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:08:35.401365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:25.386615 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:25.386534 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:25.481184 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:25.481152 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:30.432883 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:30.432839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:32.887350 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:32.887315 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:33.080442 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.080410 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t"] Sep 16 20:09:33.080675 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.080663 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerName="main" Sep 16 20:09:33.080675 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.080676 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerName="main" Sep 16 20:09:33.080763 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.080722 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc1fe16-93f2-47ef-9c18-2e6b24dbae47" containerName="main" Sep 16 20:09:33.083439 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.083419 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:09:33.088081 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.088059 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\"" Sep 16 20:09:33.088081 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.088077 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-hhfxw\"" Sep 16 20:09:33.088272 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.088119 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\"" Sep 16 20:09:33.088272 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.088220 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\"" Sep 16 20:09:33.088453 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.088436 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-rmb2x\"" Sep 16 20:09:33.098903 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.098879 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t"] Sep 16 20:09:33.117237 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.117202 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:33.157299 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.157209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4280ec7-e07e-4448-8e00-e93601e25695-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t\" (UID: \"b4280ec7-e07e-4448-8e00-e93601e25695\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:09:33.157299 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.157246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrx7n\" (UniqueName: \"kubernetes.io/projected/b4280ec7-e07e-4448-8e00-e93601e25695-kube-api-access-qrx7n\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t\" (UID: \"b4280ec7-e07e-4448-8e00-e93601e25695\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:09:33.257850 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.257807 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4280ec7-e07e-4448-8e00-e93601e25695-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t\" (UID: \"b4280ec7-e07e-4448-8e00-e93601e25695\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:09:33.257850 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.257848 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrx7n\" (UniqueName: \"kubernetes.io/projected/b4280ec7-e07e-4448-8e00-e93601e25695-kube-api-access-qrx7n\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t\" (UID: \"b4280ec7-e07e-4448-8e00-e93601e25695\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:09:33.260303 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.260284 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4280ec7-e07e-4448-8e00-e93601e25695-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t\" (UID: \"b4280ec7-e07e-4448-8e00-e93601e25695\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:09:33.268211 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.268189 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrx7n\" (UniqueName: \"kubernetes.io/projected/b4280ec7-e07e-4448-8e00-e93601e25695-kube-api-access-qrx7n\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t\" (UID: \"b4280ec7-e07e-4448-8e00-e93601e25695\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:09:33.393636 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.393598 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:09:33.663371 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:33.663325 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t"] Sep 16 20:09:33.666057 ip-10-0-139-17 kubenswrapper[2562]: W0916 20:09:33.666027 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4280ec7_e07e_4448_8e00_e93601e25695.slice/crio-e5f8f6f1f5eeb6d6f85f9f4ed6cf35d1c19e34d4adc244a180cf2be374585378 WatchSource:0}: Error finding container e5f8f6f1f5eeb6d6f85f9f4ed6cf35d1c19e34d4adc244a180cf2be374585378: Status 404 returned error can't find the container with id e5f8f6f1f5eeb6d6f85f9f4ed6cf35d1c19e34d4adc244a180cf2be374585378 Sep 16 20:09:34.363704 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:34.363667 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" event={"ID":"b4280ec7-e07e-4448-8e00-e93601e25695","Type":"ContainerStarted","Data":"e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e"} Sep 16 20:09:34.364095 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:34.363721 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" event={"ID":"b4280ec7-e07e-4448-8e00-e93601e25695","Type":"ContainerStarted","Data":"e5f8f6f1f5eeb6d6f85f9f4ed6cf35d1c19e34d4adc244a180cf2be374585378"} Sep 16 20:09:34.364095 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:34.363767 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:09:34.381224 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:34.381172 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" podStartSLOduration=1.381155842 podStartE2EDuration="1.381155842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:09:34.380428816 +0000 UTC m=+3882.929486089" watchObservedRunningTime="2026-09-16 20:09:34.381155842 +0000 UTC m=+3882.930213116" Sep 16 20:09:41.680479 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:41.680443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:43.256184 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:43.256148 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:51.885901 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:51.885864 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:52.144445 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:52.144354 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:09:52.147524 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:52.147501 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:09:53.200831 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:53.200787 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:09:59.160324 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:09:59.160285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:00.382304 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:00.382275 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:00.488949 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:00.488916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:00.603090 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:00.603058 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:05.368970 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:05.368942 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:10:06.593414 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:06.593379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:06.661275 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:06.661240 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:07.998507 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:07.998466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:13.404089 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:13.404052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:14.678217 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:14.678170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:14.755444 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:14.755402 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:23.186324 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:23.186282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:10:23.283523 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:10:23.283488 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:11:10.853510 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:11:10.853465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:11:10.918940 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:11:10.918901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:11:55.321066 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:11:55.321026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:11:55.340157 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:11:55.340129 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t"] Sep 16 20:11:55.369156 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:11:55.369117 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:11:55.421317 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:11:55.421286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:01.976495 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:01.976457 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:05.368737 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:05.368700 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:12:10.359902 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:10.359842 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" containerName="main" containerID="cri-o://e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e" gracePeriod=45 Sep 16 20:12:15.368670 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:15.368577 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:12:15.369092 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:15.368712 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:12:22.080576 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:22.080533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:22.190574 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:22.190537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:22.292931 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:22.292893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:22.401730 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:22.401695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:22.498911 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:22.498876 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:22.599561 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:22.599522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:24.834618 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:24.834581 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:24.935550 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:24.935514 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:25.368857 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:25.368825 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:12:25.556125 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:25.556083 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:28.275288 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:28.275252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:28.371513 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:28.371473 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:35.368623 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:35.368583 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:12:40.589602 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:40.589575 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:12:40.689897 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:40.689788 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4280ec7-e07e-4448-8e00-e93601e25695-tls-certs\") pod \"b4280ec7-e07e-4448-8e00-e93601e25695\" (UID: \"b4280ec7-e07e-4448-8e00-e93601e25695\") " Sep 16 20:12:40.690112 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:40.689938 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrx7n\" (UniqueName: \"kubernetes.io/projected/b4280ec7-e07e-4448-8e00-e93601e25695-kube-api-access-qrx7n\") pod \"b4280ec7-e07e-4448-8e00-e93601e25695\" (UID: \"b4280ec7-e07e-4448-8e00-e93601e25695\") " Sep 16 20:12:40.692180 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:40.692145 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4280ec7-e07e-4448-8e00-e93601e25695-kube-api-access-qrx7n" (OuterVolumeSpecName: "kube-api-access-qrx7n") pod "b4280ec7-e07e-4448-8e00-e93601e25695" (UID: "b4280ec7-e07e-4448-8e00-e93601e25695"). InnerVolumeSpecName "kube-api-access-qrx7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:12:40.692297 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:40.692189 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4280ec7-e07e-4448-8e00-e93601e25695-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4280ec7-e07e-4448-8e00-e93601e25695" (UID: "b4280ec7-e07e-4448-8e00-e93601e25695"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:12:40.790869 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:40.790828 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrx7n\" (UniqueName: \"kubernetes.io/projected/b4280ec7-e07e-4448-8e00-e93601e25695-kube-api-access-qrx7n\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:12:40.791074 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:40.790883 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4280ec7-e07e-4448-8e00-e93601e25695-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:12:41.061380 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.061289 2562 generic.go:356] "Generic (PLEG): container finished" podID="b4280ec7-e07e-4448-8e00-e93601e25695" containerID="e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e" exitCode=0 Sep 16 20:12:41.061380 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.061343 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" Sep 16 20:12:41.061574 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.061377 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" event={"ID":"b4280ec7-e07e-4448-8e00-e93601e25695","Type":"ContainerDied","Data":"e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e"} Sep 16 20:12:41.061574 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.061437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t" event={"ID":"b4280ec7-e07e-4448-8e00-e93601e25695","Type":"ContainerDied","Data":"e5f8f6f1f5eeb6d6f85f9f4ed6cf35d1c19e34d4adc244a180cf2be374585378"} Sep 16 20:12:41.061574 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.061459 2562 scope.go:117] "RemoveContainer" containerID="e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e" Sep 16 20:12:41.069945 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.069923 2562 scope.go:117] "RemoveContainer" containerID="e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e" Sep 16 20:12:41.070251 ip-10-0-139-17 kubenswrapper[2562]: E0916 20:12:41.070227 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e\": container with ID starting with e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e not found: ID does not exist" containerID="e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e" Sep 16 20:12:41.070337 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.070263 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e"} err="failed to get container status \"e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e\": rpc error: code = NotFound desc = could not find container \"e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e\": container with ID starting with e86995af5988631bc0fcbec3ddac5071c782628b9e661ba967f28d9d14221c2e not found: ID does not exist" Sep 16 20:12:41.084441 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.084418 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t"] Sep 16 20:12:41.091218 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.091198 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche55r8t"] Sep 16 20:12:41.962660 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:41.962626 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" path="/var/lib/kubelet/pods/b4280ec7-e07e-4448-8e00-e93601e25695/volumes" Sep 16 20:12:42.482870 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:42.482827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:45.401249 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:45.401210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:45.492251 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:45.492218 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:51.657321 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:51.657270 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:53.799227 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:53.799195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:57.402606 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:57.402573 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:12:57.472974 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:12:57.472936 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:13:05.285684 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:13:05.285649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:13:05.549444 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:13:05.549346 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:13:22.665433 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:13:22.665385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:13:41.977022 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:13:41.976960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:14:52.166116 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:14:52.166089 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:14:52.170114 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:14:52.170091 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:15:13.249868 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:13.249824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:33.324811 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:33.324722 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:33.448855 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:33.448826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:33.548370 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:33.548330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:33.658437 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:33.658406 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:33.798837 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:33.798798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:33.896024 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:33.895979 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:36.003612 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:36.003575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:36.102946 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:36.102913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:36.825534 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:36.825492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:38.989501 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:38.989472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:39.089029 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:39.088970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:39.188847 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:39.188813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:39.288554 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:39.288450 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:39.696180 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:39.696140 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:45.004508 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:45.004467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:45.076672 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:45.076635 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:46.018981 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:46.018942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:15:46.113793 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:15:46.113760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:01.436709 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:01.436673 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:13.092630 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:13.092587 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:14.817760 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:14.817721 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:14.888812 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:14.888772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:16.612865 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:16.612827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:34.777223 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:34.777185 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:34.870604 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:34.870565 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:35.093571 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:35.093533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:35.174214 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:35.174180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:38.246846 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:38.246799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:41.860565 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:41.860529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:47.728438 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:47.728352 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:16:47.810293 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:16:47.810260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:17:02.946916 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:02.946867 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:17:06.089720 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.089660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:17:06.431210 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.431129 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp"] Sep 16 20:17:06.431412 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.431401 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" containerName="main" Sep 16 20:17:06.431463 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.431413 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" containerName="main" Sep 16 20:17:06.431510 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.431485 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4280ec7-e07e-4448-8e00-e93601e25695" containerName="main" Sep 16 20:17:06.434305 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.434284 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:17:06.442946 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.442914 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\"" Sep 16 20:17:06.443095 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.442978 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\"" Sep 16 20:17:06.443095 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.442973 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-ldddp\"" Sep 16 20:17:06.443095 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.443049 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\"" Sep 16 20:17:06.443095 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.442983 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-h5w44\"" Sep 16 20:17:06.448399 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.448380 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp"] Sep 16 20:17:06.488916 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.488876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7mc\" (UniqueName: \"kubernetes.io/projected/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-kube-api-access-wr7mc\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp\" (UID: \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:17:06.489119 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.488936 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp\" (UID: \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:17:06.589977 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.589941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7mc\" (UniqueName: \"kubernetes.io/projected/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-kube-api-access-wr7mc\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp\" (UID: \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:17:06.590170 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.589990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp\" (UID: \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:17:06.592408 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.592377 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp\" (UID: \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:17:06.598422 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.598403 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7mc\" (UniqueName: \"kubernetes.io/projected/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-kube-api-access-wr7mc\") pod \"router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp\" (UID: \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:17:06.744467 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.744377 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:17:06.745505 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.745480 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:17:06.841868 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:06.841836 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:17:07.013199 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:07.013106 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp"] Sep 16 20:17:07.017445 ip-10-0-139-17 kubenswrapper[2562]: W0916 20:17:07.017412 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdbe82b_0862_4a12_ad2d_6b66cd21d88d.slice/crio-1176e80640dd42c05f43e11a36efad866785bacc73cc7534e168c4635f8b3166 WatchSource:0}: Error finding container 1176e80640dd42c05f43e11a36efad866785bacc73cc7534e168c4635f8b3166: Status 404 returned error can't find the container with id 1176e80640dd42c05f43e11a36efad866785bacc73cc7534e168c4635f8b3166 Sep 16 20:17:07.032629 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:07.032602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" event={"ID":"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d","Type":"ContainerStarted","Data":"1176e80640dd42c05f43e11a36efad866785bacc73cc7534e168c4635f8b3166"} Sep 16 20:17:08.038720 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:08.038685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" event={"ID":"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d","Type":"ContainerStarted","Data":"2c43efc69a7582db0bf22567225ec5c536ff508c24e389cadc5df28e8848f72d"} Sep 16 20:17:08.039160 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:08.038813 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:17:08.078266 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:08.078213 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" podStartSLOduration=2.078196097 podStartE2EDuration="2.078196097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:17:08.075073966 +0000 UTC m=+4336.624131240" watchObservedRunningTime="2026-09-16 20:17:08.078196097 +0000 UTC m=+4336.627253371" Sep 16 20:17:39.045561 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:17:39.045529 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:18:16.038205 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:16.038114 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:36.137252 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:36.137211 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:37.550892 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:37.550858 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:38.163211 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:38.163180 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp"] Sep 16 20:18:42.939547 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:42.939508 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:43.007655 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:43.007620 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:46.099771 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:46.099732 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:46.172475 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:46.172435 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:48.831628 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:48.831590 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:48.932573 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:48.932535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:53.181681 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:53.181631 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" podUID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" containerName="main" containerID="cri-o://2c43efc69a7582db0bf22567225ec5c536ff508c24e389cadc5df28e8848f72d" gracePeriod=45 Sep 16 20:18:54.297735 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.297703 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc"] Sep 16 20:18:54.300898 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.300881 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.304059 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.304032 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:18:54.304195 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.304081 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\"" Sep 16 20:18:54.304195 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.304130 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\"" Sep 16 20:18:54.304857 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.304834 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-wsfb2\"" Sep 16 20:18:54.304969 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.304873 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\"" Sep 16 20:18:54.304969 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.304921 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-dockercfg-fccq2\"" Sep 16 20:18:54.304969 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.304930 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\"" Sep 16 20:18:54.325750 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.325725 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc"] Sep 16 20:18:54.354905 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.354873 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:54.402774 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.402742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-model-cache\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.402945 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.402778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-kserve-provision-location\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.402945 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.402836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-home\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.402945 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.402856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4042ea-7ced-451e-9ed6-3082601234f4-tls-certs\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.402945 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.402877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-dshm\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.402945 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.402898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-tmp-dir\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.403151 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.402960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4042ea-7ced-451e-9ed6-3082601234f4-cabundle-cert\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.403151 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.402987 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72msd\" (UniqueName: \"kubernetes.io/projected/ab4042ea-7ced-451e-9ed6-3082601234f4-kube-api-access-72msd\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504382 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-model-cache\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504382 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504382 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-kserve-provision-location\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504655 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-home\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504655 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504458 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4042ea-7ced-451e-9ed6-3082601234f4-tls-certs\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504655 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-dshm\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504655 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-tmp-dir\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504655 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4042ea-7ced-451e-9ed6-3082601234f4-cabundle-cert\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504655 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72msd\" (UniqueName: \"kubernetes.io/projected/ab4042ea-7ced-451e-9ed6-3082601234f4-kube-api-access-72msd\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504940 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504800 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-model-cache\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.504940 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504833 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-kserve-provision-location\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.505032 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.504951 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-tmp-dir\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.505181 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.505152 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-home\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.505309 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.505265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4042ea-7ced-451e-9ed6-3082601234f4-cabundle-cert\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.506788 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.506765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-dshm\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.506888 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.506869 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4042ea-7ced-451e-9ed6-3082601234f4-tls-certs\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.514190 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.514167 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72msd\" (UniqueName: \"kubernetes.io/projected/ab4042ea-7ced-451e-9ed6-3082601234f4-kube-api-access-72msd\") pod \"tls-verification-test-kserve-7d4dfdccd8-hhblc\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.610999 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.610955 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:54.627708 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.627679 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:18:54.889874 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:54.889808 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc"] Sep 16 20:18:54.892044 ip-10-0-139-17 kubenswrapper[2562]: W0916 20:18:54.891998 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4042ea_7ced_451e_9ed6_3082601234f4.slice/crio-266900961a31544ecec98c99ccdf4f8609fb7c5b766a149e18793fb38e1f2b84 WatchSource:0}: Error finding container 266900961a31544ecec98c99ccdf4f8609fb7c5b766a149e18793fb38e1f2b84: Status 404 returned error can't find the container with id 266900961a31544ecec98c99ccdf4f8609fb7c5b766a149e18793fb38e1f2b84 Sep 16 20:18:55.461182 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:55.461144 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" event={"ID":"ab4042ea-7ced-451e-9ed6-3082601234f4","Type":"ContainerStarted","Data":"0ec1b1f21b8a5624fdd37f1ae07cadccbc64563ce226f3311ae838ee1b986263"} Sep 16 20:18:55.461581 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:55.461224 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" event={"ID":"ab4042ea-7ced-451e-9ed6-3082601234f4","Type":"ContainerStarted","Data":"266900961a31544ecec98c99ccdf4f8609fb7c5b766a149e18793fb38e1f2b84"} Sep 16 20:18:55.461581 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:55.461286 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:18:55.462397 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:55.462379 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 20:18:59.045129 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:18:59.045077 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" podUID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:19:00.484638 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:00.484604 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" event={"ID":"ab4042ea-7ced-451e-9ed6-3082601234f4","Type":"ContainerStarted","Data":"c7841718b0cbc7de2eb980e8e24e14a94b2bc757fff9b765fdaa05fda0edb9a1"} Sep 16 20:19:02.494428 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:02.494340 2562 generic.go:356] "Generic (PLEG): container finished" podID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerID="c7841718b0cbc7de2eb980e8e24e14a94b2bc757fff9b765fdaa05fda0edb9a1" exitCode=0 Sep 16 20:19:02.494428 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:02.494376 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" event={"ID":"ab4042ea-7ced-451e-9ed6-3082601234f4","Type":"ContainerDied","Data":"c7841718b0cbc7de2eb980e8e24e14a94b2bc757fff9b765fdaa05fda0edb9a1"} Sep 16 20:19:09.045816 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:09.045761 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" podUID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:19:12.510592 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:12.510556 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:19:19.045077 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:19.045033 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" podUID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:19:19.045501 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:19.045170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:19:29.630062 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:29.630002 2562 generic.go:356] "Generic (PLEG): container finished" podID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" containerID="2c43efc69a7582db0bf22567225ec5c536ff508c24e389cadc5df28e8848f72d" exitCode=0 Sep 16 20:19:29.630508 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:29.630069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" event={"ID":"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d","Type":"ContainerDied","Data":"2c43efc69a7582db0bf22567225ec5c536ff508c24e389cadc5df28e8848f72d"} Sep 16 20:19:30.044072 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.043968 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" podUID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.33:9003\" within 1s: context deadline exceeded" Sep 16 20:19:30.044072 ip-10-0-139-17 kubenswrapper[2562]: W0916 20:19:30.044065 2562 logging.go:55] [core] [Channel #257 SubChannel #258]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.33:9003", ServerName: "10.132.0.33:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.33:9003: operation was canceled" Sep 16 20:19:30.204207 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.204183 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:19:30.321396 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.321368 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-tls-certs\") pod \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\" (UID: \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\") " Sep 16 20:19:30.321483 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.321411 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr7mc\" (UniqueName: \"kubernetes.io/projected/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-kube-api-access-wr7mc\") pod \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\" (UID: \"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d\") " Sep 16 20:19:30.323599 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.323572 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" (UID: "4bdbe82b-0862-4a12-ad2d-6b66cd21d88d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:19:30.323685 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.323631 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-kube-api-access-wr7mc" (OuterVolumeSpecName: "kube-api-access-wr7mc") pod "4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" (UID: "4bdbe82b-0862-4a12-ad2d-6b66cd21d88d"). InnerVolumeSpecName "kube-api-access-wr7mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:19:30.422000 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.421956 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:19:30.422000 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.421990 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7mc\" (UniqueName: \"kubernetes.io/projected/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d-kube-api-access-wr7mc\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:19:30.636023 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.635980 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" event={"ID":"ab4042ea-7ced-451e-9ed6-3082601234f4","Type":"ContainerStarted","Data":"ff36ab9f5e6e584a6426345641ce57b6a2012fac2488981af4ee45f07ee95a25"} Sep 16 20:19:30.637400 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.637375 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" event={"ID":"4bdbe82b-0862-4a12-ad2d-6b66cd21d88d","Type":"ContainerDied","Data":"1176e80640dd42c05f43e11a36efad866785bacc73cc7534e168c4635f8b3166"} Sep 16 20:19:30.637515 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.637413 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp" Sep 16 20:19:30.637515 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.637432 2562 scope.go:117] "RemoveContainer" containerID="2c43efc69a7582db0bf22567225ec5c536ff508c24e389cadc5df28e8848f72d" Sep 16 20:19:30.677061 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.676987 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podStartSLOduration=2.220103287 podStartE2EDuration="36.676970383s" totalImagesPullingTime="32.291479334s" totalInitContainerRuntime="2.165387762s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:18:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:19:30.674167338 +0000 UTC m=+4479.223224614" watchObservedRunningTime="2026-09-16 20:19:30.676970383 +0000 UTC m=+4479.226027657" Sep 16 20:19:30.691928 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.691899 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp"] Sep 16 20:19:30.695143 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.695108 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-85f8df497dzwnp"] Sep 16 20:19:30.742770 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.742739 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt"] Sep 16 20:19:30.743008 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.742997 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" containerName="main" Sep 16 20:19:30.743071 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.743024 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" containerName="main" Sep 16 20:19:30.743107 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.743079 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" containerName="main" Sep 16 20:19:30.747249 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.747231 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:19:30.752225 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.752203 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\"" Sep 16 20:19:30.752357 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.752309 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-ldddp\"" Sep 16 20:19:30.752837 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.752814 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\"" Sep 16 20:19:30.753286 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.753271 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-h5w44\"" Sep 16 20:19:30.753375 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.753309 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\"" Sep 16 20:19:30.759530 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.759509 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt"] Sep 16 20:19:30.927166 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.927076 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt\" (UID: \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:19:30.927325 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:30.927188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66n2g\" (UniqueName: \"kubernetes.io/projected/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-kube-api-access-66n2g\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt\" (UID: \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:19:31.019750 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.019717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:19:31.028639 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.028608 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66n2g\" (UniqueName: \"kubernetes.io/projected/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-kube-api-access-66n2g\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt\" (UID: \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:19:31.028749 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.028660 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt\" (UID: \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:19:31.031979 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.031955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt\" (UID: \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:19:31.037999 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.037977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66n2g\" (UniqueName: \"kubernetes.io/projected/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-kube-api-access-66n2g\") pod \"router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt\" (UID: \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:19:31.057436 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.057404 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:19:31.338807 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.338772 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt"] Sep 16 20:19:31.344059 ip-10-0-139-17 kubenswrapper[2562]: W0916 20:19:31.344025 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786f4798_bc8f_4dd9_a5bd_1ae56f21b7bc.slice/crio-5445d8bc9cafd07a8e02a84399bedc9556d2dac61da19dd6da6e05e061429672 WatchSource:0}: Error finding container 5445d8bc9cafd07a8e02a84399bedc9556d2dac61da19dd6da6e05e061429672: Status 404 returned error can't find the container with id 5445d8bc9cafd07a8e02a84399bedc9556d2dac61da19dd6da6e05e061429672 Sep 16 20:19:31.644243 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.644149 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" event={"ID":"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc","Type":"ContainerStarted","Data":"f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14"} Sep 16 20:19:31.644243 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.644203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" event={"ID":"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc","Type":"ContainerStarted","Data":"5445d8bc9cafd07a8e02a84399bedc9556d2dac61da19dd6da6e05e061429672"} Sep 16 20:19:31.682821 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.682761 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" podStartSLOduration=1.682739011 podStartE2EDuration="1.682739011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:19:31.681145928 +0000 UTC m=+4480.230203204" watchObservedRunningTime="2026-09-16 20:19:31.682739011 +0000 UTC m=+4480.231796286" Sep 16 20:19:31.962705 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:31.962622 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bdbe82b-0862-4a12-ad2d-6b66cd21d88d" path="/var/lib/kubelet/pods/4bdbe82b-0862-4a12-ad2d-6b66cd21d88d/volumes" Sep 16 20:19:32.648836 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:32.648807 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:19:34.611735 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:34.611700 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:19:34.612343 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:34.611780 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:19:34.612415 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:34.612350 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:19:44.611674 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:44.611568 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:19:52.192194 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:52.192164 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:19:52.195268 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:52.195244 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:19:54.611752 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:19:54.611705 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:20:03.655536 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:03.655495 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:20:04.611876 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:04.611834 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:20:14.612147 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:14.612095 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:20:17.075629 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:17.075590 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:20:19.643683 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:19.643640 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:20:19.667460 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:19.667417 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt"] Sep 16 20:20:19.735640 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:19.735577 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:20:23.655335 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:23.655298 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:20:24.612201 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:24.612138 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:20:33.655458 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:33.655412 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:20:34.611803 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:34.611755 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:20:34.686949 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:34.686883 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerName="main" containerID="cri-o://f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14" gracePeriod=45 Sep 16 20:20:43.654923 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:43.654876 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:20:43.655310 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:43.655004 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:20:44.612293 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:44.612233 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:20:53.655140 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:53.655096 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:20:54.611584 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:20:54.611539 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:21:03.655481 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:03.655442 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:21:04.611744 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:04.611691 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:21:04.937906 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:04.937878 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:21:04.977387 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:04.977351 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66n2g\" (UniqueName: \"kubernetes.io/projected/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-kube-api-access-66n2g\") pod \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\" (UID: \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\") " Sep 16 20:21:04.977549 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:04.977440 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-tls-certs\") pod \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\" (UID: \"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc\") " Sep 16 20:21:04.979382 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:04.979350 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" (UID: "786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:21:04.979532 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:04.979497 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-kube-api-access-66n2g" (OuterVolumeSpecName: "kube-api-access-66n2g") pod "786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" (UID: "786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc"). InnerVolumeSpecName "kube-api-access-66n2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:21:05.039312 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.039277 2562 generic.go:356] "Generic (PLEG): container finished" podID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerID="f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14" exitCode=0 Sep 16 20:21:05.039484 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.039324 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" Sep 16 20:21:05.039484 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.039325 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" event={"ID":"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc","Type":"ContainerDied","Data":"f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14"} Sep 16 20:21:05.039484 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.039460 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt" event={"ID":"786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc","Type":"ContainerDied","Data":"5445d8bc9cafd07a8e02a84399bedc9556d2dac61da19dd6da6e05e061429672"} Sep 16 20:21:05.039609 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.039494 2562 scope.go:117] "RemoveContainer" containerID="f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14" Sep 16 20:21:05.054648 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.054623 2562 scope.go:117] "RemoveContainer" containerID="f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14" Sep 16 20:21:05.054986 ip-10-0-139-17 kubenswrapper[2562]: E0916 20:21:05.054959 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14\": container with ID starting with f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14 not found: ID does not exist" containerID="f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14" Sep 16 20:21:05.055070 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.055001 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14"} err="failed to get container status \"f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14\": rpc error: code = NotFound desc = could not find container \"f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14\": container with ID starting with f8b641cd52180b6538139227e99aa16b7d1f2b617427d392978f02684505ac14 not found: ID does not exist" Sep 16 20:21:05.068545 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.068517 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt"] Sep 16 20:21:05.072663 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.072638 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-858d6c586lgnbt"] Sep 16 20:21:05.078695 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.078674 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66n2g\" (UniqueName: \"kubernetes.io/projected/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-kube-api-access-66n2g\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:21:05.078780 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.078698 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:21:05.962521 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:05.962485 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" path="/var/lib/kubelet/pods/786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc/volumes" Sep 16 20:21:14.611914 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:14.611807 2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8001/health\": dial tcp 10.132.0.34:8001: connect: connection refused" Sep 16 20:21:24.627165 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:24.627131 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:21:24.638731 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:24.638704 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:21:24.975506 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:24.975407 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:21:25.081464 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:25.081427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:21:29.932681 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:29.932637 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:21:46.561600 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:46.561568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:21:49.440800 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:49.440697 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:21:49.442836 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:49.442810 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc"] Sep 16 20:21:49.510878 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:49.510843 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:21:51.508580 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:51.508544 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:21:52.763336 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:52.763302 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:21:58.938074 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:21:58.938039 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:04.464804 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:04.464752 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" containerID="cri-o://ff36ab9f5e6e584a6426345641ce57b6a2012fac2488981af4ee45f07ee95a25" gracePeriod=45 Sep 16 20:22:05.081559 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.081520 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="llm-d-routing-sidecar" containerID="cri-o://0ec1b1f21b8a5624fdd37f1ae07cadccbc64563ce226f3311ae838ee1b986263" gracePeriod=45 Sep 16 20:22:05.276321 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.276290 2562 generic.go:356] "Generic (PLEG): container finished" podID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerID="ff36ab9f5e6e584a6426345641ce57b6a2012fac2488981af4ee45f07ee95a25" exitCode=0 Sep 16 20:22:05.276321 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.276317 2562 generic.go:356] "Generic (PLEG): container finished" podID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerID="0ec1b1f21b8a5624fdd37f1ae07cadccbc64563ce226f3311ae838ee1b986263" exitCode=0 Sep 16 20:22:05.276508 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.276347 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" event={"ID":"ab4042ea-7ced-451e-9ed6-3082601234f4","Type":"ContainerDied","Data":"ff36ab9f5e6e584a6426345641ce57b6a2012fac2488981af4ee45f07ee95a25"} Sep 16 20:22:05.276508 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.276387 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" event={"ID":"ab4042ea-7ced-451e-9ed6-3082601234f4","Type":"ContainerDied","Data":"0ec1b1f21b8a5624fdd37f1ae07cadccbc64563ce226f3311ae838ee1b986263"} Sep 16 20:22:05.327622 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.327596 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:22:05.352716 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.352689 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4042ea-7ced-451e-9ed6-3082601234f4-cabundle-cert\") pod \"ab4042ea-7ced-451e-9ed6-3082601234f4\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " Sep 16 20:22:05.352874 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.352734 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-dshm\") pod \"ab4042ea-7ced-451e-9ed6-3082601234f4\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " Sep 16 20:22:05.352874 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.352756 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-model-cache\") pod \"ab4042ea-7ced-451e-9ed6-3082601234f4\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " Sep 16 20:22:05.352874 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.352798 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-kserve-provision-location\") pod \"ab4042ea-7ced-451e-9ed6-3082601234f4\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " Sep 16 20:22:05.352874 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.352832 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72msd\" (UniqueName: \"kubernetes.io/projected/ab4042ea-7ced-451e-9ed6-3082601234f4-kube-api-access-72msd\") pod \"ab4042ea-7ced-451e-9ed6-3082601234f4\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " Sep 16 20:22:05.352874 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.352861 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-tmp-dir\") pod \"ab4042ea-7ced-451e-9ed6-3082601234f4\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " Sep 16 20:22:05.353433 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.352911 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4042ea-7ced-451e-9ed6-3082601234f4-tls-certs\") pod \"ab4042ea-7ced-451e-9ed6-3082601234f4\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " Sep 16 20:22:05.353433 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.352976 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-home\") pod \"ab4042ea-7ced-451e-9ed6-3082601234f4\" (UID: \"ab4042ea-7ced-451e-9ed6-3082601234f4\") " Sep 16 20:22:05.353433 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.353065 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab4042ea-7ced-451e-9ed6-3082601234f4" (UID: "ab4042ea-7ced-451e-9ed6-3082601234f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.353433 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.353148 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4042ea-7ced-451e-9ed6-3082601234f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab4042ea-7ced-451e-9ed6-3082601234f4" (UID: "ab4042ea-7ced-451e-9ed6-3082601234f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:22:05.353433 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.353285 2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4042ea-7ced-451e-9ed6-3082601234f4-cabundle-cert\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.353433 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.353305 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-model-cache\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.355708 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.354261 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-home" (OuterVolumeSpecName: "home") pod "ab4042ea-7ced-451e-9ed6-3082601234f4" (UID: "ab4042ea-7ced-451e-9ed6-3082601234f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.356586 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.356561 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4042ea-7ced-451e-9ed6-3082601234f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab4042ea-7ced-451e-9ed6-3082601234f4" (UID: "ab4042ea-7ced-451e-9ed6-3082601234f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:22:05.356819 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.356788 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-dshm" (OuterVolumeSpecName: "dshm") pod "ab4042ea-7ced-451e-9ed6-3082601234f4" (UID: "ab4042ea-7ced-451e-9ed6-3082601234f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.357133 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.357092 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4042ea-7ced-451e-9ed6-3082601234f4-kube-api-access-72msd" (OuterVolumeSpecName: "kube-api-access-72msd") pod "ab4042ea-7ced-451e-9ed6-3082601234f4" (UID: "ab4042ea-7ced-451e-9ed6-3082601234f4"). InnerVolumeSpecName "kube-api-access-72msd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:22:05.370956 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.370922 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab4042ea-7ced-451e-9ed6-3082601234f4" (UID: "ab4042ea-7ced-451e-9ed6-3082601234f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.376036 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.375996 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab4042ea-7ced-451e-9ed6-3082601234f4" (UID: "ab4042ea-7ced-451e-9ed6-3082601234f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.454312 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.454261 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4042ea-7ced-451e-9ed6-3082601234f4-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.454312 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.454306 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-home\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.454312 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.454314 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-dshm\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.454312 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.454323 2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-kserve-provision-location\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.454312 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.454331 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72msd\" (UniqueName: \"kubernetes.io/projected/ab4042ea-7ced-451e-9ed6-3082601234f4-kube-api-access-72msd\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.454615 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:05.454340 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4042ea-7ced-451e-9ed6-3082601234f4-tmp-dir\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:22:06.282126 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:06.282043 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" event={"ID":"ab4042ea-7ced-451e-9ed6-3082601234f4","Type":"ContainerDied","Data":"266900961a31544ecec98c99ccdf4f8609fb7c5b766a149e18793fb38e1f2b84"} Sep 16 20:22:06.282126 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:06.282057 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc" Sep 16 20:22:06.282126 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:06.282103 2562 scope.go:117] "RemoveContainer" containerID="ff36ab9f5e6e584a6426345641ce57b6a2012fac2488981af4ee45f07ee95a25" Sep 16 20:22:06.289834 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:06.289814 2562 scope.go:117] "RemoveContainer" containerID="c7841718b0cbc7de2eb980e8e24e14a94b2bc757fff9b765fdaa05fda0edb9a1" Sep 16 20:22:06.296807 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:06.296788 2562 scope.go:117] "RemoveContainer" containerID="0ec1b1f21b8a5624fdd37f1ae07cadccbc64563ce226f3311ae838ee1b986263" Sep 16 20:22:06.302989 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:06.302968 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc"] Sep 16 20:22:06.307714 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:06.307689 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-7d4dfdccd8-hhblc"] Sep 16 20:22:07.747801 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:07.747767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:07.962603 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:07.962574 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" path="/var/lib/kubelet/pods/ab4042ea-7ced-451e-9ed6-3082601234f4/volumes" Sep 16 20:22:09.053405 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:09.053359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:13.731943 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:13.731907 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:13.815385 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:13.815355 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:39.721178 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:39.721135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:39.839201 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:39.839170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:46.314689 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:46.314595 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:49.226315 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:49.226268 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:49.385556 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:49.385521 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:52.042945 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:52.042913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:22:52.144195 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:22:52.144152 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:23:10.615322 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:23:10.615283 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:23:10.922224 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:23:10.922141 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:23:26.314335 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:23:26.314282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:23:41.689356 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:23:41.689318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:23:55.336130 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:23:55.336073 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:24:46.621198 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:46.621159 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:24:46.726296 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:46.726256 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:24:52.213707 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:52.213671 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:24:52.218076 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:52.218055 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7hqb_f96bb20f-d067-49b4-b9b3-d5bdbddb07bf/ovn-acl-logging/0.log" Sep 16 20:24:53.111436 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:53.111397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:24:54.990869 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:54.990834 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:24:55.193322 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193292 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp"] Sep 16 20:24:55.193581 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193566 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="llm-d-routing-sidecar" Sep 16 20:24:55.193629 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193584 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="llm-d-routing-sidecar" Sep 16 20:24:55.193629 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193596 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="storage-initializer" Sep 16 20:24:55.193629 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193602 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="storage-initializer" Sep 16 20:24:55.193629 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193614 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" Sep 16 20:24:55.193629 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193619 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" Sep 16 20:24:55.193629 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193627 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerName="main" Sep 16 20:24:55.193806 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193634 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerName="main" Sep 16 20:24:55.193806 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193686 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="llm-d-routing-sidecar" Sep 16 20:24:55.193806 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193695 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="786f4798-bc8f-4dd9-a5bd-1ae56f21b7bc" containerName="main" Sep 16 20:24:55.193806 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.193700 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4042ea-7ced-451e-9ed6-3082601234f4" containerName="main" Sep 16 20:24:55.196449 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.196426 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:24:55.199318 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.199295 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\"" Sep 16 20:24:55.200343 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.200320 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-f6z2c\"" Sep 16 20:24:55.200472 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.200346 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-x7n4c\"" Sep 16 20:24:55.200472 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.200396 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\"" Sep 16 20:24:55.200472 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.200422 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\"" Sep 16 20:24:55.211511 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.211489 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp"] Sep 16 20:24:55.226212 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.226183 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:24:55.244290 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.244216 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5df\" (UniqueName: \"kubernetes.io/projected/a5620cef-efe1-4c00-a0ed-3553b2a4d745-kube-api-access-rf5df\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp\" (UID: \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:24:55.244290 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.244272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5620cef-efe1-4c00-a0ed-3553b2a4d745-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp\" (UID: \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:24:55.344796 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.344761 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5df\" (UniqueName: \"kubernetes.io/projected/a5620cef-efe1-4c00-a0ed-3553b2a4d745-kube-api-access-rf5df\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp\" (UID: \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:24:55.344965 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.344807 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5620cef-efe1-4c00-a0ed-3553b2a4d745-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp\" (UID: \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:24:55.347378 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.347355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5620cef-efe1-4c00-a0ed-3553b2a4d745-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp\" (UID: \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:24:55.353286 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.353264 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5df\" (UniqueName: \"kubernetes.io/projected/a5620cef-efe1-4c00-a0ed-3553b2a4d745-kube-api-access-rf5df\") pod \"rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp\" (UID: \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:24:55.506461 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.506352 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:24:55.802642 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.802546 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp"] Sep 16 20:24:55.806412 ip-10-0-139-17 kubenswrapper[2562]: W0916 20:24:55.806362 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5620cef_efe1_4c00_a0ed_3553b2a4d745.slice/crio-7f938b78e0139a9fa478d9a84ecb86618996de9f8e8e8f5ee740132707854ff4 WatchSource:0}: Error finding container 7f938b78e0139a9fa478d9a84ecb86618996de9f8e8e8f5ee740132707854ff4: Status 404 returned error can't find the container with id 7f938b78e0139a9fa478d9a84ecb86618996de9f8e8e8f5ee740132707854ff4 Sep 16 20:24:55.924891 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.924845 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" event={"ID":"a5620cef-efe1-4c00-a0ed-3553b2a4d745","Type":"ContainerStarted","Data":"e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1"} Sep 16 20:24:55.925158 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.924916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" event={"ID":"a5620cef-efe1-4c00-a0ed-3553b2a4d745","Type":"ContainerStarted","Data":"7f938b78e0139a9fa478d9a84ecb86618996de9f8e8e8f5ee740132707854ff4"} Sep 16 20:24:55.925158 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.925067 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:24:55.955045 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:55.951232 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" podStartSLOduration=0.95121261 podStartE2EDuration="951.21261ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:24:55.950102049 +0000 UTC m=+4804.499159320" watchObservedRunningTime="2026-09-16 20:24:55.95121261 +0000 UTC m=+4804.500269885" Sep 16 20:24:59.366612 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:59.366577 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:24:59.474555 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:24:59.474517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:25:10.911758 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:25:10.911719 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:25:26.930528 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:25:26.930494 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:25:41.497795 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:25:41.497763 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:25:47.497526 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:25:47.497410 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:25:47.576403 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:25:47.576364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:02.556685 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:02.556649 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp"] Sep 16 20:26:02.558830 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:02.558793 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:02.628597 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:02.628552 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:06.931073 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:06.931037 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:26:16.931146 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:16.931104 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:26:17.577293 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:17.577226 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerName="main" containerID="cri-o://e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1" gracePeriod=45 Sep 16 20:26:26.931246 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:26.931206 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:26:26.931716 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:26.931328 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:26:36.930808 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:36.930765 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:26:46.931034 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:46.930961 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:26:47.830627 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:47.830596 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:26:47.878803 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:47.878769 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf5df\" (UniqueName: \"kubernetes.io/projected/a5620cef-efe1-4c00-a0ed-3553b2a4d745-kube-api-access-rf5df\") pod \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\" (UID: \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\") " Sep 16 20:26:47.878994 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:47.878826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5620cef-efe1-4c00-a0ed-3553b2a4d745-tls-certs\") pod \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\" (UID: \"a5620cef-efe1-4c00-a0ed-3553b2a4d745\") " Sep 16 20:26:47.881051 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:47.881022 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5620cef-efe1-4c00-a0ed-3553b2a4d745-kube-api-access-rf5df" (OuterVolumeSpecName: "kube-api-access-rf5df") pod "a5620cef-efe1-4c00-a0ed-3553b2a4d745" (UID: "a5620cef-efe1-4c00-a0ed-3553b2a4d745"). InnerVolumeSpecName "kube-api-access-rf5df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:26:47.881168 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:47.881064 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5620cef-efe1-4c00-a0ed-3553b2a4d745-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5620cef-efe1-4c00-a0ed-3553b2a4d745" (UID: "a5620cef-efe1-4c00-a0ed-3553b2a4d745"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:26:47.979271 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:47.979245 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5df\" (UniqueName: \"kubernetes.io/projected/a5620cef-efe1-4c00-a0ed-3553b2a4d745-kube-api-access-rf5df\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:26:47.979271 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:47.979271 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5620cef-efe1-4c00-a0ed-3553b2a4d745-tls-certs\") on node \"ip-10-0-139-17.ec2.internal\" DevicePath \"\"" Sep 16 20:26:48.348058 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:48.348026 2562 generic.go:356] "Generic (PLEG): container finished" podID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerID="e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1" exitCode=0 Sep 16 20:26:48.348229 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:48.348091 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" Sep 16 20:26:48.348229 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:48.348104 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" event={"ID":"a5620cef-efe1-4c00-a0ed-3553b2a4d745","Type":"ContainerDied","Data":"e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1"} Sep 16 20:26:48.348229 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:48.348162 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp" event={"ID":"a5620cef-efe1-4c00-a0ed-3553b2a4d745","Type":"ContainerDied","Data":"7f938b78e0139a9fa478d9a84ecb86618996de9f8e8e8f5ee740132707854ff4"} Sep 16 20:26:48.348229 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:48.348184 2562 scope.go:117] "RemoveContainer" containerID="e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1" Sep 16 20:26:48.356406 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:48.356386 2562 scope.go:117] "RemoveContainer" containerID="e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1" Sep 16 20:26:48.356686 ip-10-0-139-17 kubenswrapper[2562]: E0916 20:26:48.356664 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1\": container with ID starting with e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1 not found: ID does not exist" containerID="e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1" Sep 16 20:26:48.356764 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:48.356696 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1"} err="failed to get container status \"e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1\": rpc error: code = NotFound desc = could not find container \"e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1\": container with ID starting with e69198bde4e95e4f1f5135ee9b18cb1791289c7ae23fd8408b17477390456ca1 not found: ID does not exist" Sep 16 20:26:48.364455 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:48.364430 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp"] Sep 16 20:26:48.371887 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:48.371862 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7bcdd447bc-wtlcp"] Sep 16 20:26:49.545795 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:49.545758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:49.961892 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:49.961858 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" path="/var/lib/kubelet/pods/a5620cef-efe1-4c00-a0ed-3553b2a4d745/volumes" Sep 16 20:26:50.553638 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:50.553602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:50.907441 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:50.907404 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:50.967246 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:50.967186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:53.537753 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:53.537720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:53.638185 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:53.638146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:55.698679 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:55.698642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:26:56.602824 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:26:56.602783 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:01.457207 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:01.457167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:01.523562 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:01.523524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:02.298045 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:02.297993 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:02.356367 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:02.356329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:17.026387 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:17.026291 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-t8llt_6e84ed16-bdae-4d25-a906-dc6785378da2/discovery/0.log" Sep 16 20:27:17.965838 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:17.965791 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-t8llt_6e84ed16-bdae-4d25-a906-dc6785378da2/discovery/0.log" Sep 16 20:27:19.354742 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:19.354714 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-7l5bs_66a682f5-23a0-4f9d-88b5-1937b74f3b3d/manager/0.log" Sep 16 20:27:19.366887 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:19.366861 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-qlvpv_3f1b44d4-bd18-4c46-a596-9910d87ce890/kuadrant-console-plugin/0.log" Sep 16 20:27:24.074172 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:24.074131 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:24.227407 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:24.227375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:24.328693 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:24.328603 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:27.326206 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:27.326177 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hjmsb_8a31ef62-788d-40d4-b735-08587d7e341d/global-pull-secret-syncer/0.log" Sep 16 20:27:27.387360 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:27.387330 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" Sep 16 20:27:27.574028 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:27.573973 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-17.ec2.internal_322d0d38fd19fa48489dd88803493fc9/haproxy/0.log" Sep 16 20:27:30.110901 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:30.110864 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:30.173867 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:30.173832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pjtv9_ffe5e8fe-18fc-43f5-8f36-23b3e89025fa/konnectivity-agent/0.log" limit=1024 Sep 16 20:27:32.792959 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:32.792921 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-7l5bs_66a682f5-23a0-4f9d-88b5-1937b74f3b3d/manager/0.log" Sep 16 20:27:32.822334 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:32.822308 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-qlvpv_3f1b44d4-bd18-4c46-a596-9910d87ce890/kuadrant-console-plugin/0.log" Sep 16 20:27:34.972913 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:34.972891 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ff7n8_cccb3134-b55b-44de-8f31-6f7182e05922/node-exporter/0.log" Sep 16 20:27:35.057386 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:35.057353 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ff7n8_cccb3134-b55b-44de-8f31-6f7182e05922/kube-rbac-proxy/0.log" Sep 16 20:27:35.095396 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:35.095354 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ff7n8_cccb3134-b55b-44de-8f31-6f7182e05922/init-textfile/0.log" Sep 16 20:27:37.585776 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.585739 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9"] Sep 16 20:27:37.586171 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.585999 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerName="main" Sep 16 20:27:37.586171 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.586023 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerName="main" Sep 16 20:27:37.586171 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.586072 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5620cef-efe1-4c00-a0ed-3553b2a4d745" containerName="main" Sep 16 20:27:37.588933 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.588912 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.591218 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.591199 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vcftn\"/\"default-dockercfg-hfqdr\"" Sep 16 20:27:37.592146 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.592128 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vcftn\"/\"kube-root-ca.crt\"" Sep 16 20:27:37.592210 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.592148 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vcftn\"/\"openshift-service-ca.crt\"" Sep 16 20:27:37.598238 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.598214 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9"] Sep 16 20:27:37.659268 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.659233 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-podres\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.659268 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.659274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-sys\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.659522 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.659305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6jmw\" (UniqueName: \"kubernetes.io/projected/2410f61c-4817-4f00-aebd-11c35b4f3e0c-kube-api-access-g6jmw\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.659522 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.659350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-lib-modules\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.659522 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.659418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-proc\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.760290 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.760249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-proc\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.760492 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.760310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-podres\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.760492 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.760333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-sys\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.760492 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.760365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6jmw\" (UniqueName: \"kubernetes.io/projected/2410f61c-4817-4f00-aebd-11c35b4f3e0c-kube-api-access-g6jmw\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.760492 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.760410 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-lib-modules\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.760492 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.760434 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-sys\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.760492 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.760411 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-proc\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.760492 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.760475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-podres\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.760758 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.760540 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2410f61c-4817-4f00-aebd-11c35b4f3e0c-lib-modules\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.768764 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.768731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6jmw\" (UniqueName: \"kubernetes.io/projected/2410f61c-4817-4f00-aebd-11c35b4f3e0c-kube-api-access-g6jmw\") pod \"perf-node-gather-daemonset-v9gp9\" (UID: \"2410f61c-4817-4f00-aebd-11c35b4f3e0c\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:37.899197 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:37.899113 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:38.167396 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:38.167338 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9"] Sep 16 20:27:38.169965 ip-10-0-139-17 kubenswrapper[2562]: W0916 20:27:38.169935 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2410f61c_4817_4f00_aebd_11c35b4f3e0c.slice/crio-723e6afe3c1229ae6e65dee19e2e5629602437df0569e7c02c1845be524739c1 WatchSource:0}: Error finding container 723e6afe3c1229ae6e65dee19e2e5629602437df0569e7c02c1845be524739c1: Status 404 returned error can't find the container with id 723e6afe3c1229ae6e65dee19e2e5629602437df0569e7c02c1845be524739c1 Sep 16 20:27:38.547134 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:38.547044 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" event={"ID":"2410f61c-4817-4f00-aebd-11c35b4f3e0c","Type":"ContainerStarted","Data":"fe80bbc3f95f9c9a7521354b1d78a2a292d63ff7a91d2810c6b558474b085791"} Sep 16 20:27:38.547134 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:38.547097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" event={"ID":"2410f61c-4817-4f00-aebd-11c35b4f3e0c","Type":"ContainerStarted","Data":"723e6afe3c1229ae6e65dee19e2e5629602437df0569e7c02c1845be524739c1"} Sep 16 20:27:38.547322 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:38.547150 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:39.897285 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:39.897249 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fjz6k_fb283ce1-2ad2-4616-b24f-9560df64c816/dns/0.log" Sep 16 20:27:39.924123 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:39.924100 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fjz6k_fb283ce1-2ad2-4616-b24f-9560df64c816/kube-rbac-proxy/0.log" Sep 16 20:27:40.119154 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:40.119125 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vq45m_5bf475c5-b68d-4235-b53e-ff2420b1e556/dns-node-resolver/0.log" Sep 16 20:27:40.732187 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:40.732159 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ft7jx_9f54592e-2056-4dd4-969c-eaeec3227210/node-ca/0.log" Sep 16 20:27:41.901351 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:41.901315 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-t8llt_6e84ed16-bdae-4d25-a906-dc6785378da2/discovery/0.log" Sep 16 20:27:42.527171 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:42.527137 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w5tdv_2a0bfe9b-6279-4c42-9568-812c00a190e8/serve-healthcheck-canary/0.log" Sep 16 20:27:43.019994 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:43.019963 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f6sfj_94ea5312-94ba-403e-88cf-ad816a2daad9/kube-rbac-proxy/0.log" Sep 16 20:27:43.040619 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:43.040594 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f6sfj_94ea5312-94ba-403e-88cf-ad816a2daad9/exporter/0.log" Sep 16 20:27:43.064193 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:43.064168 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f6sfj_94ea5312-94ba-403e-88cf-ad816a2daad9/extractor/0.log" Sep 16 20:27:44.560142 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:44.560117 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" Sep 16 20:27:44.576367 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:44.576316 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-v9gp9" podStartSLOduration=7.576302567 podStartE2EDuration="7.576302567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:27:38.586724682 +0000 UTC m=+4967.135781955" watchObservedRunningTime="2026-09-16 20:27:44.576302567 +0000 UTC m=+4973.125359842" Sep 16 20:27:46.280633 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:46.280605 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9498697bd-7v7vb_1aae3cfa-bcd8-4102-95f6-855769c0d67e/manager/0.log" Sep 16 20:27:47.686414 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:47.686325 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-t8llt_6e84ed16-bdae-4d25-a906-dc6785378da2/discovery/0.log" Sep 16 20:27:49.145650 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:49.145584 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-t8llt_6e84ed16-bdae-4d25-a906-dc6785378da2/discovery/0.log" Sep 16 20:27:54.958713 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:54.958674 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5tk6q_f66049e9-573e-4aa1-add4-b930be25fb66/kube-multus/0.log" Sep 16 20:27:55.136421 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:55.136392 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2hzb_f01f26bd-0336-4941-bc54-62734d65f013/kube-multus-additional-cni-plugins/0.log" Sep 16 20:27:55.159909 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:55.159875 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2hzb_f01f26bd-0336-4941-bc54-62734d65f013/egress-router-binary-copy/0.log" Sep 16 20:27:55.179655 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:55.179631 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2hzb_f01f26bd-0336-4941-bc54-62734d65f013/cni-plugins/0.log" Sep 16 20:27:55.198934 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:55.198911 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2hzb_f01f26bd-0336-4941-bc54-62734d65f013/bond-cni-plugin/0.log" Sep 16 20:27:55.222608 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:55.222527 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2hzb_f01f26bd-0336-4941-bc54-62734d65f013/routeoverride-cni/0.log" Sep 16 20:27:55.244722 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:55.244683 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2hzb_f01f26bd-0336-4941-bc54-62734d65f013/whereabouts-cni-bincopy/0.log" Sep 16 20:27:55.264795 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:55.264768 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2hzb_f01f26bd-0336-4941-bc54-62734d65f013/whereabouts-cni/0.log" Sep 16 20:27:55.591906 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:55.591873 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mzvlv_10b511a5-508f-4ce7-988a-7abc0d8e70da/network-metrics-daemon/0.log" Sep 16 20:27:55.611831 ip-10-0-139-17 kubenswrapper[2562]: I0916 20:27:55.611805 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mzvlv_10b511a5-508f-4ce7-988a-7abc0d8e70da/kube-rbac-proxy/0.log"