Sep 08 13:16:12.331171 ip-10-0-133-71 systemd[1]: Starting Kubernetes Kubelet... Sep 08 13:16:12.761768 ip-10-0-133-71 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 13:16:12.761768 ip-10-0-133-71 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 13:16:12.761768 ip-10-0-133-71 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 13:16:12.761768 ip-10-0-133-71 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 13:16:12.761768 ip-10-0-133-71 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 13:16:12.764135 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.764044 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 13:16:12.768280 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768264 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768281 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768286 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768289 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768292 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768295 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768298 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768300 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768304 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768307 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768310 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768312 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768315 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768318 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768320 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768323 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 13:16:12.768318 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768326 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768335 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768338 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768341 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768343 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768346 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768349 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768352 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768354 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768357 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768360 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768363 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768366 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768369 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768372 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768374 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768377 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768380 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768383 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 13:16:12.768719 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768386 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768388 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768391 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768393 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768396 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768398 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768401 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768404 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768406 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768408 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768411 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768413 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768416 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768418 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768421 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768425 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768429 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768432 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768435 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768438 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 13:16:12.769279 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768440 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768443 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768446 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768448 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768451 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768455 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768458 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768461 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768464 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768466 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768469 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768472 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768475 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768477 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768480 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768484 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768487 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768490 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768494 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 13:16:12.769798 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768496 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768499 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768502 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768504 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768507 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768509 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768512 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768514 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768518 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768520 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768523 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768526 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768528 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768531 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768533 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768536 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768538 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768541 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768543 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768546 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 13:16:12.770287 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768548 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768551 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768554 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768556 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.768559 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769276 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769282 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769286 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769289 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769292 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769295 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769298 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769301 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769304 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769306 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769309 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769311 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769314 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769317 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769319 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 13:16:12.770772 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769323 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769325 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769328 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769331 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769334 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769336 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769339 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769343 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769350 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769353 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769355 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769358 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769361 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769363 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769366 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769369 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769371 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769374 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769377 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 13:16:12.771325 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769379 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769382 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769384 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769387 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769390 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769393 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769396 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769398 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769401 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769405 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769409 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769412 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769415 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769418 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769421 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769424 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769426 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769429 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769432 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769435 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 13:16:12.771821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769438 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769440 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769443 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769446 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769448 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769451 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769453 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769456 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769458 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769461 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769463 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769465 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769468 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769470 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769473 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769475 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769477 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769481 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769484 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769486 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 13:16:12.772320 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769489 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769492 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769494 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769498 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769502 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769505 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769508 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769511 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769514 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769517 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769520 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769523 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769526 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769528 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769531 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769534 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769536 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769538 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769541 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 13:16:12.772808 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769543 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769546 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769548 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769551 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769554 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769556 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769964 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769971 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769974 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769977 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769984 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769987 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769990 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769993 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769996 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.769999 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770002 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770005 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770008 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 13:16:12.773284 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770011 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770013 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770016 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770018 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770021 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770024 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770026 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770029 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770031 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770034 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770036 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770039 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770041 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770044 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770047 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770050 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770053 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770056 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770058 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770060 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 13:16:12.773758 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770063 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770066 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770068 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770073 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770076 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770079 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770082 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770084 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770087 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770089 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770092 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770095 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770098 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770101 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770103 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770106 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770108 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770111 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770114 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770116 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 13:16:12.774264 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770119 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770121 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770124 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770126 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770129 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770131 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770134 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770136 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770139 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770141 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770144 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770146 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770150 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770152 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770154 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770159 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770161 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770167 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770170 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770172 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 13:16:12.774748 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770175 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770178 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770180 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770183 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770185 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770188 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770191 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770193 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770195 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770198 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770200 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770203 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770206 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770208 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770211 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770213 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770216 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770219 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770221 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770224 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 13:16:12.775245 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770226 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770229 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770231 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770234 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770236 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.770239 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770309 2554 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770345 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770354 2554 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770359 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770364 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770368 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770372 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770376 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770380 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770382 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770386 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770389 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770392 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770395 2554 flags.go:64] FLAG: --cgroup-root="" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770398 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770401 2554 flags.go:64] FLAG: --client-ca-file="" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770403 2554 flags.go:64] FLAG: --cloud-provider="external" Sep 08 13:16:12.775730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770406 2554 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770411 2554 flags.go:64] FLAG: --cluster-domain="" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770414 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770417 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770420 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770424 2554 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770428 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770431 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770434 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770437 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770440 2554 flags.go:64] FLAG: --contention-profiling="false" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770443 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770446 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770449 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770452 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770456 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770461 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770464 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770467 2554 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770470 2554 flags.go:64] FLAG: --enable-server="true" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770477 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770481 2554 flags.go:64] FLAG: --event-burst="100" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770484 2554 flags.go:64] FLAG: --event-qps="50" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770487 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 13:16:12.776314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770490 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770493 2554 flags.go:64] FLAG: --eviction-hard="" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770496 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770500 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770503 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770506 2554 flags.go:64] FLAG: --eviction-soft="" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770508 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770511 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770514 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770517 2554 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770520 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770523 2554 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770525 2554 flags.go:64] FLAG: --feature-gates="" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770529 2554 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770532 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770535 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770539 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770542 2554 flags.go:64] FLAG: --healthz-port="10248" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770545 2554 flags.go:64] FLAG: --help="false" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770547 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-133-71.ec2.internal" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770550 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770554 2554 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770556 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770560 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 13:16:12.776941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770565 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770568 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770571 2554 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770573 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770576 2554 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770585 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770588 2554 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770591 2554 flags.go:64] FLAG: --kube-reserved="" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770594 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770597 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770600 2554 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770603 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770605 2554 flags.go:64] FLAG: --lock-file="" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770608 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770611 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770614 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770620 2554 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770622 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770625 2554 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770629 2554 flags.go:64] FLAG: --logging-format="text" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770631 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770635 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770637 2554 flags.go:64] FLAG: --manifest-url="" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770640 2554 flags.go:64] FLAG: --manifest-url-header="" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770644 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 13:16:12.777533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770647 2554 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770651 2554 flags.go:64] FLAG: --max-pods="110" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770654 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770657 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770660 2554 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770663 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770666 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770671 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770674 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770682 2554 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770685 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770688 2554 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770691 2554 flags.go:64] FLAG: --pod-cidr="" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770699 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770704 2554 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770707 2554 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770710 2554 flags.go:64] FLAG: --pods-per-core="0" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770713 2554 flags.go:64] FLAG: --port="10250" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770716 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770718 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-008811d3788588ca4" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770721 2554 flags.go:64] FLAG: --qos-reserved="" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770725 2554 flags.go:64] FLAG: --read-only-port="10255" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770727 2554 flags.go:64] FLAG: --register-node="true" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770730 2554 flags.go:64] FLAG: --register-with-taints="" Sep 08 13:16:12.778166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770734 2554 flags.go:64] FLAG: --registry-burst="10" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770737 2554 flags.go:64] FLAG: --registry-qps="5" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770739 2554 flags.go:64] FLAG: --reserved-cpus="" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770742 2554 flags.go:64] FLAG: --reserved-memory="" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770745 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770748 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770751 2554 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770754 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770757 2554 flags.go:64] FLAG: --runonce="false" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770759 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770762 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770765 2554 flags.go:64] FLAG: --seccomp-default="false" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770768 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770771 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770774 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770778 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770782 2554 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770785 2554 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770788 2554 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770791 2554 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770794 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770797 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770805 2554 flags.go:64] FLAG: --system-cgroups="" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770808 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770813 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 13:16:12.778756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770816 2554 flags.go:64] FLAG: --tls-cert-file="" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770819 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770826 2554 flags.go:64] FLAG: --tls-min-version="" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770829 2554 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770832 2554 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770835 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770838 2554 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770841 2554 flags.go:64] FLAG: --v="2" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770844 2554 flags.go:64] FLAG: --version="false" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770848 2554 flags.go:64] FLAG: --vmodule="" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770852 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.770855 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771005 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771009 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771012 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771015 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771018 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771021 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771023 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771026 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771029 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771032 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771035 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 13:16:12.779353 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771039 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771042 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771044 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771047 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771050 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771052 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771055 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771059 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771062 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771064 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771067 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771069 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771072 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771075 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771077 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771080 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771082 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771085 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771088 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771090 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 13:16:12.779933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771093 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771095 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771099 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771103 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771106 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771109 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771111 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771114 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771117 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771119 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771122 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771125 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771129 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771132 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771136 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771140 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771143 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771145 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771148 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 13:16:12.780418 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771150 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771153 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771156 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771158 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771161 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771165 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771168 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771171 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771174 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771177 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771179 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771182 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771185 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771187 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771190 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771192 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771195 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771198 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771200 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 13:16:12.781005 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771203 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771206 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771209 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771211 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771214 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771216 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771220 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771223 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771225 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771228 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771230 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771233 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771236 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771238 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771241 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771244 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771246 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771249 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771251 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 13:16:12.781469 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771253 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771256 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771258 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771261 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771264 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771266 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771269 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771271 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771274 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771277 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.771280 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 13:16:12.781965 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.771285 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 13:16:12.782266 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.782017 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 13:16:12.782266 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.782039 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 13:16:12.782980 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.782969 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 13:16:12.783016 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.782984 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 13:16:12.783048 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783042 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 13:16:12.783048 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783047 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783050 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783053 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783057 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783060 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783063 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783066 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783068 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783071 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783074 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783076 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783079 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783082 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783085 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783087 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783090 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783092 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783095 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783097 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 13:16:12.783099 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783100 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783104 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783109 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783112 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783115 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783118 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783120 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783123 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783125 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783129 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783132 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783135 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783138 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783140 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783143 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783146 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783148 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783151 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783153 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783156 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 13:16:12.783619 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783159 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783161 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783163 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783166 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783170 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783173 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783176 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783179 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783182 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783184 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783187 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783190 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783193 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783195 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783198 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783201 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783204 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783207 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783209 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 13:16:12.784389 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783212 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783214 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783217 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783220 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783224 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783227 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783230 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783233 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783235 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783238 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783240 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783243 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783245 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783248 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783250 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783253 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783255 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783258 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783261 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783264 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 13:16:12.784964 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783267 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783269 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783272 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783274 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783277 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783281 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783284 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783286 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783289 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783291 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783294 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783296 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783299 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783301 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783304 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783306 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783309 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783312 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783314 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 13:16:12.785460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783317 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.783322 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783696 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783720 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783727 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783732 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783737 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783741 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783746 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783750 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783755 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783760 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783767 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783774 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 13:16:12.785980 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783780 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783791 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783796 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783801 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783805 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783809 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783813 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783818 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783822 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783826 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783830 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783835 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783838 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783842 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783852 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783856 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783875 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783880 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783885 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 13:16:12.786321 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783889 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783893 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783897 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783901 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783906 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783911 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783914 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783923 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783927 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783932 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783936 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783940 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783944 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783949 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783953 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783957 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783961 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783966 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783969 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783974 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 13:16:12.786820 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783982 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783987 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783991 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783995 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.783999 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784004 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784008 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784012 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784016 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784021 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784026 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784030 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784035 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784045 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784049 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784057 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784063 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784067 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784071 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 13:16:12.787331 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784076 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784080 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784084 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784089 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784093 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784097 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784107 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784111 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784114 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784118 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784122 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784126 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784149 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784153 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784158 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784162 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784166 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784170 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784179 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784184 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 13:16:12.787883 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784188 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784193 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784197 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784201 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784207 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784211 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784218 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784223 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:12.784229 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.784237 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.784489 2554 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.786403 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.787491 2554 server.go:1019] "Starting client certificate rotation" Sep 08 13:16:12.788374 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.787605 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 13:16:12.788823 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.788557 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 13:16:12.790355 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.790339 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 13:16:12.791576 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.791558 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 13:16:12.803225 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.803206 2554 log.go:25] "Validated CRI v1 runtime API" Sep 08 13:16:12.808170 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.808153 2554 log.go:25] "Validated CRI v1 image API" Sep 08 13:16:12.809055 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.809022 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 13:16:12.812685 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.812655 2554 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8fefb6f2-4f75-4828-84f6-685941684fd3:/dev/nvme0n1p3 94e3d184-30f2-4688-87e5-97801aebb455:/dev/nvme0n1p4] Sep 08 13:16:12.812775 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.812683 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 13:16:12.817434 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.817415 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 13:16:12.818139 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.818014 2554 manager.go:216] Machine: {Timestamp:2026-09-08 13:16:12.816671699 +0000 UTC m=+0.373167425 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116662 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22b8a017ef985f2900ae2cb586a43c SystemUUID:ec22b8a0-17ef-985f-2900-ae2cb586a43c BootID:1f39190b-1e1c-466d-a2ab-7da4b9feeb31 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:35:d7:e8:9f:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:35:d7:e8:9f:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:5e:59:70:0a:19 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 13:16:12.818139 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.818134 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 13:16:12.818296 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.818254 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 13:16:12.821711 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.821670 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 13:16:12.821962 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.821717 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 13:16:12.822652 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.822638 2554 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 13:16:12.822714 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.822657 2554 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 13:16:12.822714 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.822668 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 13:16:12.824125 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.824112 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 13:16:12.825189 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.825177 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 13:16:12.829606 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.829592 2554 state_mem.go:36] "Initialized new in-memory state store" Sep 08 13:16:12.829760 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.829750 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 13:16:12.829794 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.829780 2554 kubelet.go:486] "Attempting to sync node with API server" Sep 08 13:16:12.829825 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.829814 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 13:16:12.829825 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.829824 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 13:16:12.829895 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.829834 2554 kubelet.go:398] "Adding apiserver pod source" Sep 08 13:16:12.829933 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.829918 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 13:16:12.830391 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.830378 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 13:16:12.830436 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.830404 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 13:16:12.834582 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.834562 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 13:16:12.837540 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.837523 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 13:16:12.837628 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.837561 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 13:16:12.838939 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.838925 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 13:16:12.839012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.838947 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 13:16:12.839012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.838959 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 13:16:12.839012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.838966 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 13:16:12.839012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.838978 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 13:16:12.839012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.838987 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 13:16:12.839012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.839006 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 13:16:12.839012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.839014 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 13:16:12.839241 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.839024 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 13:16:12.839241 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.839032 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 13:16:12.839241 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.839045 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 13:16:12.839241 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.839053 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 13:16:12.839791 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.839780 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 13:16:12.839844 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.839793 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 13:16:12.842247 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.842225 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 13:16:12.843035 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.843009 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 13:16:12.843035 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.843015 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 13:16:12.843797 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.843783 2554 server.go:1268] "Started kubelet" Sep 08 13:16:12.843888 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.843849 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 13:16:12.843942 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.843855 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 13:16:12.843980 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.843945 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 13:16:12.844816 ip-10-0-133-71 systemd[1]: Started Kubernetes Kubelet. Sep 08 13:16:12.845357 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.845127 2554 server.go:310] "Adding debug handlers to kubelet server" Sep 08 13:16:12.845619 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.845495 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 13:16:12.848133 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.848113 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 13:16:12.854365 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.854319 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 13:16:12.854926 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.854902 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 13:16:12.855113 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.855088 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:12.855703 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.855685 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 13:16:12.855841 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.855821 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 13:16:12.855959 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.855706 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 13:16:12.855959 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.855904 2554 factory.go:55] Registering systemd factory Sep 08 13:16:12.855959 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.855917 2554 factory.go:223] Registration of the systemd container factory successfully Sep 08 13:16:12.856088 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.856008 2554 reconstruct.go:97] "Volume reconstruction finished" Sep 08 13:16:12.856088 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.856016 2554 reconciler.go:29] "Reconciler: start to sync state" Sep 08 13:16:12.856748 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.856602 2554 factory.go:153] Registering CRI-O factory Sep 08 13:16:12.856835 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.856752 2554 factory.go:223] Registration of the crio container factory successfully Sep 08 13:16:12.856901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.856832 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 13:16:12.856901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.856879 2554 factory.go:103] Registering Raw factory Sep 08 13:16:12.856901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.856899 2554 manager.go:1196] Started watching for new ooms in manager Sep 08 13:16:12.857350 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.857337 2554 manager.go:349] Starting recovery of all containers Sep 08 13:16:12.863038 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.863012 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 13:16:12.863535 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.863508 2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 13:16:12.864707 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.863565 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d35adad081cb75 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-08 13:16:12.843756405 +0000 UTC m=+0.400252131,LastTimestamp:2026-09-08 13:16:12.843756405 +0000 UTC m=+0.400252131,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}" Sep 08 13:16:12.869068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.869048 2554 manager.go:354] Recovery completed Sep 08 13:16:12.874094 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.874079 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 13:16:12.875794 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.875770 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory" Sep 08 13:16:12.875877 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.875808 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 13:16:12.875877 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.875818 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID" Sep 08 13:16:12.876451 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.876434 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 13:16:12.876451 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.876450 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 13:16:12.876578 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.876455 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 13:16:12.876578 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.876468 2554 state_mem.go:36] "Initialized new in-memory state store" Sep 08 13:16:12.877227 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.877209 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory" Sep 08 13:16:12.877284 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.877245 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 13:16:12.877284 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.877259 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID" Sep 08 13:16:12.878338 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.878257 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d35adad26aa7b9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-08 13:16:12.875794361 +0000 UTC m=+0.432290090,LastTimestamp:2026-09-08 13:16:12.875794361 +0000 UTC m=+0.432290090,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}" Sep 08 13:16:12.878663 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.878653 2554 policy_none.go:49] "None policy: Start" Sep 08 13:16:12.878713 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.878668 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 13:16:12.878713 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.878678 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 13:16:12.880702 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.880686 2554 policy_none.go:47] "Start" Sep 08 13:16:12.886654 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.886568 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d35adad26aefb9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-08 13:16:12.875812793 +0000 UTC m=+0.432308522,LastTimestamp:2026-09-08 13:16:12.875812793 +0000 UTC m=+0.432308522,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}" Sep 08 13:16:12.900595 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.900524 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d35adad26b1540 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-08 13:16:12.8758224 +0000 UTC m=+0.432318125,LastTimestamp:2026-09-08 13:16:12.8758224 +0000 UTC m=+0.432318125,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.910237 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-71.ec2.internal.18d35adad26aa7b9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d35adad26aa7b9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-08 13:16:12.875794361 +0000 UTC m=+0.432290090,LastTimestamp:2026-09-08 13:16:12.877227912 +0000 UTC m=+0.433723694,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.918429 2554 manager.go:337] "Starting Device Plugin manager" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.918462 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.918473 2554 server.go:87] "Starting device plugin registration server" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.918834 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.918847 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.918979 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.919078 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.919086 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.919722 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.919764 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:12.934476 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.922327 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-71.ec2.internal.18d35adad26aefb9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d35adad26aefb9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-08 13:16:12.875812793 +0000 UTC m=+0.432308522,LastTimestamp:2026-09-08 13:16:12.877252282 +0000 UTC m=+0.433748015,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}" Sep 08 13:16:12.943324 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.943257 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-71.ec2.internal.18d35adad26b1540\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d35adad26b1540 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-08 13:16:12.8758224 +0000 UTC m=+0.432318125,LastTimestamp:2026-09-08 13:16:12.877264799 +0000 UTC m=+0.433760526,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}" Sep 08 13:16:12.946346 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.946327 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zzk88" Sep 08 13:16:12.955111 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.955087 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zzk88" Sep 08 13:16:12.955400 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.955335 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d35adad51c9ac2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-08 13:16:12.921010882 +0000 UTC m=+0.477506599,LastTimestamp:2026-09-08 13:16:12.921010882 +0000 UTC m=+0.477506599,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}" Sep 08 13:16:12.957410 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.957374 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 13:16:12.958674 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.958655 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 13:16:12.958753 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.958704 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 13:16:12.958753 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.958732 2554 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 13:16:12.958816 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:12.958795 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 13:16:12.967478 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:12.967458 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 13:16:13.019302 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.019211 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 13:16:13.020284 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.020264 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory" Sep 08 13:16:13.020388 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.020297 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 13:16:13.020388 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.020307 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID" Sep 08 13:16:13.020388 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.020330 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.030342 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.030318 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.030457 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:13.030344 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-71.ec2.internal\": node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:13.058880 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.058844 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal"] Sep 08 13:16:13.059027 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.058920 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 13:16:13.059027 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:13.058944 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:13.060451 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.060439 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory" Sep 08 13:16:13.060511 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.060465 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 13:16:13.060511 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.060476 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID" Sep 08 13:16:13.062894 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.062876 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 13:16:13.063029 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.062949 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.063029 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.062982 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 13:16:13.063625 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.063607 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory" Sep 08 13:16:13.063734 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.063639 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 13:16:13.063734 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.063649 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID" Sep 08 13:16:13.063734 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.063609 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory" Sep 08 13:16:13.063734 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.063723 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 13:16:13.063847 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.063756 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID" Sep 08 13:16:13.065942 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.065929 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.065998 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.065953 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 13:16:13.066597 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.066583 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory" Sep 08 13:16:13.066654 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.066611 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 13:16:13.066654 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.066623 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID" Sep 08 13:16:13.090451 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:13.090429 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-71.ec2.internal\" not found" node="ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.094812 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:13.094798 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-71.ec2.internal\" not found" node="ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.156610 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.156584 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8629079a46d2d7de6ec5fcd3cb9f06dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"8629079a46d2d7de6ec5fcd3cb9f06dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.156610 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.156613 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8629079a46d2d7de6ec5fcd3cb9f06dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"8629079a46d2d7de6ec5fcd3cb9f06dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.156776 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.156654 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d4cca95c4b58a556d25d9ef4b9a311c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-71.ec2.internal\" (UID: \"4d4cca95c4b58a556d25d9ef4b9a311c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.159666 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:13.159636 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:13.257179 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.257146 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8629079a46d2d7de6ec5fcd3cb9f06dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"8629079a46d2d7de6ec5fcd3cb9f06dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.257272 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.257185 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d4cca95c4b58a556d25d9ef4b9a311c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-71.ec2.internal\" (UID: \"4d4cca95c4b58a556d25d9ef4b9a311c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.257272 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.257205 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8629079a46d2d7de6ec5fcd3cb9f06dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"8629079a46d2d7de6ec5fcd3cb9f06dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.257272 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.257234 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8629079a46d2d7de6ec5fcd3cb9f06dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"8629079a46d2d7de6ec5fcd3cb9f06dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.257272 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.257250 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8629079a46d2d7de6ec5fcd3cb9f06dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"8629079a46d2d7de6ec5fcd3cb9f06dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.257272 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:13.257250 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d4cca95c4b58a556d25d9ef4b9a311c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-71.ec2.internal\" (UID: \"4d4cca95c4b58a556d25d9ef4b9a311c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" Sep 08 13:16:13.260227 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:13.260211 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:14.584844 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:14.584819 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:14.616019 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:14.615978 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" Sep 08 13:16:14.622047 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:14.622031 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" Sep 08 13:16:14.685271 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:14.685234 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:14.785812 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:14.785774 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:14.886449 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:14.886350 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:14.986946 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:14.986912 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:15.011124 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.011084 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 13:16:15.011272 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.011255 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 13:16:15.087181 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:15.087157 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:15.184054 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.183957 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:11:14 +0000 UTC" deadline="2028-04-23 05:55:24.356680016 +0000 UTC" Sep 08 13:16:15.184054 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.183991 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14224h39m9.172693307s" Sep 08 13:16:15.187728 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:15.187713 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:15.288277 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:15.288235 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:15.296405 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.296385 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 13:16:15.297312 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.297293 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory" Sep 08 13:16:15.297412 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.297325 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 13:16:15.297412 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.297335 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID" Sep 08 13:16:15.363346 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.363316 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 13:16:15.389092 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:15.389065 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:15.489677 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:15.489601 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found" Sep 08 13:16:15.548524 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.548494 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 13:16:15.579933 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.579902 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" Sep 08 13:16:15.595931 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.595907 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 13:16:15.597672 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.597655 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" Sep 08 13:16:15.609443 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.609427 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 13:16:15.641611 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.641585 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 13:16:15.744933 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:15.744868 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8629079a46d2d7de6ec5fcd3cb9f06dc.slice/crio-e8b4237a97f80db567d7a03cd4ee37ef4af53e8a69436d0c8e508f5b27e8965f WatchSource:0}: Error finding container e8b4237a97f80db567d7a03cd4ee37ef4af53e8a69436d0c8e508f5b27e8965f: Status 404 returned error can't find the container with id e8b4237a97f80db567d7a03cd4ee37ef4af53e8a69436d0c8e508f5b27e8965f Sep 08 13:16:15.749398 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.749381 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 13:16:15.819247 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:15.819212 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 13:16:15.955303 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:15.955278 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4cca95c4b58a556d25d9ef4b9a311c.slice/crio-02840746e97a1ec213db82b7550947006ab8a5d1ae877990bc4b0c698b755d15 WatchSource:0}: Error finding container 02840746e97a1ec213db82b7550947006ab8a5d1ae877990bc4b0c698b755d15: Status 404 returned error can't find the container with id 02840746e97a1ec213db82b7550947006ab8a5d1ae877990bc4b0c698b755d15 Sep 08 13:16:16.055205 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.055170 2554 apiserver.go:52] "Watching apiserver" Sep 08 13:16:16.061716 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.061675 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 13:16:16.062022 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.062002 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-hgc6w","kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks","openshift-cluster-node-tuning-operator/tuned-vq4jw","openshift-ovn-kubernetes/ovnkube-node-xxmjp","openshift-dns/node-resolver-hp94q","openshift-image-registry/node-ca-rgj2t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal","openshift-multus/multus-additional-cni-plugins-v6zgx","openshift-multus/multus-zf6qs","openshift-multus/network-metrics-daemon-sghnw","openshift-network-diagnostics/network-check-target-q8rk6","openshift-network-operator/iptables-alerter-kvrtd","kube-system/global-pull-secret-syncer-vl8zn"] Sep 08 13:16:16.064727 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.064712 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.066698 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.066665 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.067080 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.067063 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 13:16:16.067332 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.067307 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 13:16:16.067420 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.067350 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 13:16:16.067623 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.067611 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zrsfk\"" Sep 08 13:16:16.068967 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.068927 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.069063 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.068993 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 13:16:16.069444 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.069426 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 13:16:16.069444 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.069439 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 13:16:16.069576 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.069490 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-blhbl\"" Sep 08 13:16:16.071244 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.071028 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.071244 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.071172 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 13:16:16.071244 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.071202 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 13:16:16.071632 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.071618 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4j6gb\"" Sep 08 13:16:16.071711 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.071657 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 13:16:16.073121 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.073105 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 13:16:16.073469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.073453 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.073589 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.073524 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 13:16:16.073989 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.073975 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4b7wf\"" Sep 08 13:16:16.075705 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.075660 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 13:16:16.075855 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.075840 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 13:16:16.075926 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.075890 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.076628 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.076610 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 13:16:16.076733 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.076629 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 13:16:16.076733 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.076644 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9vlj5\"" Sep 08 13:16:16.076733 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.076674 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 13:16:16.076898 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.076759 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 13:16:16.078084 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.078066 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 13:16:16.078172 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.078088 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 13:16:16.078346 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.078175 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-295rk\"" Sep 08 13:16:16.078346 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.078178 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.078660 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.078503 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 13:16:16.080524 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.080505 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 13:16:16.080524 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.080532 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 13:16:16.080697 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.080567 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:16.080697 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.080655 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnkm7\"" Sep 08 13:16:16.081000 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.080981 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 13:16:16.081000 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.080995 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 13:16:16.081115 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.081052 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 13:16:16.082947 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.082932 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.083028 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.083007 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:16.083612 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.083597 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tc4xg\"" Sep 08 13:16:16.084001 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.083987 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 13:16:16.084651 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.084636 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 13:16:16.087462 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.086756 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.088712 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.088694 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:16.088825 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.088803 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:16.088984 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.088969 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-run-netns\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.089026 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.088996 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-log-socket\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.089026 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089014 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-cni-netd\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.089103 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089043 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-ovn-node-metrics-cert\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.089103 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089091 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2xw\" (UniqueName: \"kubernetes.io/projected/dc121a5f-2eee-4ca2-a204-fcefee21d9ff-kube-api-access-cm2xw\") pod \"node-ca-rgj2t\" (UID: \"dc121a5f-2eee-4ca2-a204-fcefee21d9ff\") " pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.089219 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089134 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.089219 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089175 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-run\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.089219 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089194 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-sys\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.089363 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089237 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-var-lib-kubelet\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.089363 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089286 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/311ed079-bea5-4b44-b238-fca763ef6788-system-cni-dir\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.089363 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089321 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gtf\" (UniqueName: \"kubernetes.io/projected/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-kube-api-access-x2gtf\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.089363 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089355 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-sysctl-conf\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.089545 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089381 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-systemd-units\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.089545 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089438 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-env-overrides\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.089545 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089475 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/311ed079-bea5-4b44-b238-fca763ef6788-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.089847 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089825 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/311ed079-bea5-4b44-b238-fca763ef6788-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.089933 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089867 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-registration-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.089933 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089887 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-etc-openvswitch\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.089933 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089902 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-node-log\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.089933 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089921 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/98e75a43-fd1d-4b84-b0b6-56cdd64db693-iptables-alerter-script\") pod \"iptables-alerter-kvrtd\" (UID: \"98e75a43-fd1d-4b84-b0b6-56cdd64db693\") " pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.090118 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089954 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/311ed079-bea5-4b44-b238-fca763ef6788-cni-binary-copy\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.090118 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.089985 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-socket-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.090118 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090009 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-sys-fs\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.090118 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090025 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 13:16:16.090118 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090043 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-run-ovn\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090118 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090066 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4wkmk\"" Sep 08 13:16:16.090118 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090069 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090118 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090109 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-systemd\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.090387 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090130 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-slash\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090387 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090151 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-run-openvswitch\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090387 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090213 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmf74\" (UniqueName: \"kubernetes.io/projected/98e75a43-fd1d-4b84-b0b6-56cdd64db693-kube-api-access-xmf74\") pod \"iptables-alerter-kvrtd\" (UID: \"98e75a43-fd1d-4b84-b0b6-56cdd64db693\") " pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.090387 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090253 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgg8c\" (UniqueName: \"kubernetes.io/projected/311ed079-bea5-4b44-b238-fca763ef6788-kube-api-access-bgg8c\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.090387 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090281 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.090387 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090314 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-sysctl-d\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.090387 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090331 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19f87d80-4c14-4f36-bd27-852618164729-hosts-file\") pod \"node-resolver-hp94q\" (UID: \"19f87d80-4c14-4f36-bd27-852618164729\") " pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.090387 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090345 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc121a5f-2eee-4ca2-a204-fcefee21d9ff-serviceca\") pod \"node-ca-rgj2t\" (UID: \"dc121a5f-2eee-4ca2-a204-fcefee21d9ff\") " pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.090387 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090365 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-lib-modules\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.090631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090399 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9v4t\" (UniqueName: \"kubernetes.io/projected/19f87d80-4c14-4f36-bd27-852618164729-kube-api-access-b9v4t\") pod \"node-resolver-hp94q\" (UID: \"19f87d80-4c14-4f36-bd27-852618164729\") " pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.090631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090427 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-kubernetes\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.090631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090449 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-run-systemd\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090492 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090532 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/311ed079-bea5-4b44-b238-fca763ef6788-cnibin\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.090631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090558 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-device-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.090631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090578 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af8db987-1561-4e57-87f0-1573ffc18033-etc-tuned\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.090631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090592 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-kubelet\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090605 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-ovnkube-config\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090631 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-host\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090665 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-var-lib-openvswitch\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090718 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-ovnkube-script-lib\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090763 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-sysconfig\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090789 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af8db987-1561-4e57-87f0-1573ffc18033-tmp\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090808 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/944df588-8dba-4a35-80af-b21714f5ca22-agent-certs\") pod \"konnectivity-agent-hgc6w\" (UID: \"944df588-8dba-4a35-80af-b21714f5ca22\") " pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090822 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7bcr\" (UniqueName: \"kubernetes.io/projected/af8db987-1561-4e57-87f0-1573ffc18033-kube-api-access-c7bcr\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090848 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-cni-bin\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090875 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvq7j\" (UniqueName: \"kubernetes.io/projected/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-kube-api-access-dvq7j\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090893 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98e75a43-fd1d-4b84-b0b6-56cdd64db693-host-slash\") pod \"iptables-alerter-kvrtd\" (UID: \"98e75a43-fd1d-4b84-b0b6-56cdd64db693\") " pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090917 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/311ed079-bea5-4b44-b238-fca763ef6788-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.090944 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090938 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/944df588-8dba-4a35-80af-b21714f5ca22-konnectivity-ca\") pod \"konnectivity-agent-hgc6w\" (UID: \"944df588-8dba-4a35-80af-b21714f5ca22\") " pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:16.091347 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090952 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-modprobe-d\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.091347 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.090978 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f87d80-4c14-4f36-bd27-852618164729-tmp-dir\") pod \"node-resolver-hp94q\" (UID: \"19f87d80-4c14-4f36-bd27-852618164729\") " pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.091347 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.091007 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc121a5f-2eee-4ca2-a204-fcefee21d9ff-host\") pod \"node-ca-rgj2t\" (UID: \"dc121a5f-2eee-4ca2-a204-fcefee21d9ff\") " pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.091347 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.091228 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:16.091347 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.091304 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:16.098375 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.098354 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 13:16:16.119145 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.119120 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6z76p" Sep 08 13:16:16.128870 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.128846 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6z76p" Sep 08 13:16:16.180888 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.180857 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 13:16:16.186825 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.186784 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" event={"ID":"4d4cca95c4b58a556d25d9ef4b9a311c","Type":"ContainerStarted","Data":"02840746e97a1ec213db82b7550947006ab8a5d1ae877990bc4b0c698b755d15"} Sep 08 13:16:16.187923 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.187886 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" event={"ID":"8629079a46d2d7de6ec5fcd3cb9f06dc","Type":"ContainerStarted","Data":"e8b4237a97f80db567d7a03cd4ee37ef4af53e8a69436d0c8e508f5b27e8965f"} Sep 08 13:16:16.191181 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191159 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-sysctl-conf\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.191294 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191191 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-systemd-units\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.191294 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191214 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-env-overrides\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.191294 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191238 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/311ed079-bea5-4b44-b238-fca763ef6788-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.191294 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191264 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/311ed079-bea5-4b44-b238-fca763ef6788-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.191294 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191270 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-systemd-units\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.191294 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191290 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-registration-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191317 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-cni-dir\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191334 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-sysctl-conf\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191343 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-etc-openvswitch\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191372 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-node-log\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191389 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-registration-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191396 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/98e75a43-fd1d-4b84-b0b6-56cdd64db693-iptables-alerter-script\") pod \"iptables-alerter-kvrtd\" (UID: \"98e75a43-fd1d-4b84-b0b6-56cdd64db693\") " pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191412 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-etc-openvswitch\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191440 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/311ed079-bea5-4b44-b238-fca763ef6788-cni-binary-copy\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191451 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-node-log\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-socket-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191475 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/311ed079-bea5-4b44-b238-fca763ef6788-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191491 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-sys-fs\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191521 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-run-k8s-cni-cncf-io\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191535 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-sys-fs\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191546 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-run-ovn\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.191583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191578 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191608 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191615 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-socket-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191580 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-run-ovn\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191606 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02ad25a4-76db-4b91-b212-a701afcd24b8-kubelet-config\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191713 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-var-lib-kubelet\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191759 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6mfg\" (UniqueName: \"kubernetes.io/projected/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-kube-api-access-h6mfg\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191788 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-systemd\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191799 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-env-overrides\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191811 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-slash\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191847 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-run-openvswitch\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191862 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-slash\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191881 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-systemd\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.192177 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.191900 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/311ed079-bea5-4b44-b238-fca763ef6788-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192493 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/98e75a43-fd1d-4b84-b0b6-56cdd64db693-iptables-alerter-script\") pod \"iptables-alerter-kvrtd\" (UID: \"98e75a43-fd1d-4b84-b0b6-56cdd64db693\") " pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192501 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-run-openvswitch\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192533 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/311ed079-bea5-4b44-b238-fca763ef6788-cni-binary-copy\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192547 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmf74\" (UniqueName: \"kubernetes.io/projected/98e75a43-fd1d-4b84-b0b6-56cdd64db693-kube-api-access-xmf74\") pod \"iptables-alerter-kvrtd\" (UID: \"98e75a43-fd1d-4b84-b0b6-56cdd64db693\") " pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192565 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgg8c\" (UniqueName: \"kubernetes.io/projected/311ed079-bea5-4b44-b238-fca763ef6788-kube-api-access-bgg8c\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192581 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192597 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192611 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-hostroot\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192632 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-sysctl-d\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192643 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192649 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19f87d80-4c14-4f36-bd27-852618164729-hosts-file\") pod \"node-resolver-hp94q\" (UID: \"19f87d80-4c14-4f36-bd27-852618164729\") " pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192705 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19f87d80-4c14-4f36-bd27-852618164729-hosts-file\") pod \"node-resolver-hp94q\" (UID: \"19f87d80-4c14-4f36-bd27-852618164729\") " pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192722 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc121a5f-2eee-4ca2-a204-fcefee21d9ff-serviceca\") pod \"node-ca-rgj2t\" (UID: \"dc121a5f-2eee-4ca2-a204-fcefee21d9ff\") " pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.192747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192746 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-run-netns\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192772 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-lib-modules\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192776 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-sysctl-d\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192798 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9v4t\" (UniqueName: \"kubernetes.io/projected/19f87d80-4c14-4f36-bd27-852618164729-kube-api-access-b9v4t\") pod \"node-resolver-hp94q\" (UID: \"19f87d80-4c14-4f36-bd27-852618164729\") " pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192819 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-cnibin\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192847 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00c3d528-0dd0-4c53-99d3-181afc05cf12-cni-binary-copy\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192888 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-conf-dir\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192910 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-etc-kubernetes\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192933 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-kubernetes\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192937 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-lib-modules\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192969 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-run-systemd\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.192989 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-kubernetes\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193007 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-run-systemd\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193021 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193050 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/311ed079-bea5-4b44-b238-fca763ef6788-cnibin\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193072 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc121a5f-2eee-4ca2-a204-fcefee21d9ff-serviceca\") pod \"node-ca-rgj2t\" (UID: \"dc121a5f-2eee-4ca2-a204-fcefee21d9ff\") " pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193073 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-device-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.193257 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193099 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/311ed079-bea5-4b44-b238-fca763ef6788-cnibin\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193105 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193116 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-device-dir\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193119 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdh7m\" (UniqueName: \"kubernetes.io/projected/00c3d528-0dd0-4c53-99d3-181afc05cf12-kube-api-access-qdh7m\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193147 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193169 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af8db987-1561-4e57-87f0-1573ffc18033-etc-tuned\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193202 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-kubelet\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193280 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-kubelet\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193291 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-ovnkube-config\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193317 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-host\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193336 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-var-lib-openvswitch\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193356 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-ovnkube-script-lib\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193378 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-host\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193385 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02ad25a4-76db-4b91-b212-a701afcd24b8-dbus\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193415 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-var-lib-openvswitch\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193433 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-system-cni-dir\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193450 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-daemon-config\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.193901 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-sysconfig\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193482 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af8db987-1561-4e57-87f0-1573ffc18033-tmp\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193510 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/944df588-8dba-4a35-80af-b21714f5ca22-agent-certs\") pod \"konnectivity-agent-hgc6w\" (UID: \"944df588-8dba-4a35-80af-b21714f5ca22\") " pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193530 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193534 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-sysconfig\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193546 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7bcr\" (UniqueName: \"kubernetes.io/projected/af8db987-1561-4e57-87f0-1573ffc18033-kube-api-access-c7bcr\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193560 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-cni-bin\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193583 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvq7j\" (UniqueName: \"kubernetes.io/projected/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-kube-api-access-dvq7j\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193605 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98e75a43-fd1d-4b84-b0b6-56cdd64db693-host-slash\") pod \"iptables-alerter-kvrtd\" (UID: \"98e75a43-fd1d-4b84-b0b6-56cdd64db693\") " pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193628 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/311ed079-bea5-4b44-b238-fca763ef6788-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193651 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/944df588-8dba-4a35-80af-b21714f5ca22-konnectivity-ca\") pod \"konnectivity-agent-hgc6w\" (UID: \"944df588-8dba-4a35-80af-b21714f5ca22\") " pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193674 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-socket-dir-parent\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193714 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-modprobe-d\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193736 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f87d80-4c14-4f36-bd27-852618164729-tmp-dir\") pod \"node-resolver-hp94q\" (UID: \"19f87d80-4c14-4f36-bd27-852618164729\") " pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193762 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98e75a43-fd1d-4b84-b0b6-56cdd64db693-host-slash\") pod \"iptables-alerter-kvrtd\" (UID: \"98e75a43-fd1d-4b84-b0b6-56cdd64db693\") " pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193777 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc121a5f-2eee-4ca2-a204-fcefee21d9ff-host\") pod \"node-ca-rgj2t\" (UID: \"dc121a5f-2eee-4ca2-a204-fcefee21d9ff\") " pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193797 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-ovnkube-config\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.194777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193806 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-var-lib-cni-multus\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193813 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-cni-bin\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193773 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193868 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc121a5f-2eee-4ca2-a204-fcefee21d9ff-host\") pod \"node-ca-rgj2t\" (UID: \"dc121a5f-2eee-4ca2-a204-fcefee21d9ff\") " pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193889 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-etc-modprobe-d\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.193951 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-ovnkube-script-lib\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194004 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-run-netns\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194032 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-log-socket\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194081 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-cni-netd\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194096 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f87d80-4c14-4f36-bd27-852618164729-tmp-dir\") pod \"node-resolver-hp94q\" (UID: \"19f87d80-4c14-4f36-bd27-852618164729\") " pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194101 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-run-netns\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194105 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-ovn-node-metrics-cert\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194148 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2xw\" (UniqueName: \"kubernetes.io/projected/dc121a5f-2eee-4ca2-a204-fcefee21d9ff-kube-api-access-cm2xw\") pod \"node-ca-rgj2t\" (UID: \"dc121a5f-2eee-4ca2-a204-fcefee21d9ff\") " pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194176 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194197 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-var-lib-cni-bin\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194217 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-run-multus-certs\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194237 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-run\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194244 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/944df588-8dba-4a35-80af-b21714f5ca22-konnectivity-ca\") pod \"konnectivity-agent-hgc6w\" (UID: \"944df588-8dba-4a35-80af-b21714f5ca22\") " pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:16.195523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194255 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-sys\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194275 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-var-lib-kubelet\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194296 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/311ed079-bea5-4b44-b238-fca763ef6788-system-cni-dir\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194316 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gtf\" (UniqueName: \"kubernetes.io/projected/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-kube-api-access-x2gtf\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194384 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/311ed079-bea5-4b44-b238-fca763ef6788-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194443 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-sys\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194491 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-run\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194532 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af8db987-1561-4e57-87f0-1573ffc18033-var-lib-kubelet\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194534 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194553 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/311ed079-bea5-4b44-b238-fca763ef6788-system-cni-dir\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194579 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-log-socket\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.196469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.194612 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-host-cni-netd\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.197275 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.197248 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af8db987-1561-4e57-87f0-1573ffc18033-etc-tuned\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.197357 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.197285 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af8db987-1561-4e57-87f0-1573ffc18033-tmp\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.197488 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.197373 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-ovn-node-metrics-cert\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.197551 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.197466 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/944df588-8dba-4a35-80af-b21714f5ca22-agent-certs\") pod \"konnectivity-agent-hgc6w\" (UID: \"944df588-8dba-4a35-80af-b21714f5ca22\") " pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:16.203587 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.203565 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgg8c\" (UniqueName: \"kubernetes.io/projected/311ed079-bea5-4b44-b238-fca763ef6788-kube-api-access-bgg8c\") pod \"multus-additional-cni-plugins-v6zgx\" (UID: \"311ed079-bea5-4b44-b238-fca763ef6788\") " pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.203842 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.203815 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9v4t\" (UniqueName: \"kubernetes.io/projected/19f87d80-4c14-4f36-bd27-852618164729-kube-api-access-b9v4t\") pod \"node-resolver-hp94q\" (UID: \"19f87d80-4c14-4f36-bd27-852618164729\") " pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.204033 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.204003 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmf74\" (UniqueName: \"kubernetes.io/projected/98e75a43-fd1d-4b84-b0b6-56cdd64db693-kube-api-access-xmf74\") pod \"iptables-alerter-kvrtd\" (UID: \"98e75a43-fd1d-4b84-b0b6-56cdd64db693\") " pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.204253 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.204236 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvq7j\" (UniqueName: \"kubernetes.io/projected/2047f6b0-59c8-4ec1-bfa7-846d78da60b4-kube-api-access-dvq7j\") pod \"ovnkube-node-xxmjp\" (UID: \"2047f6b0-59c8-4ec1-bfa7-846d78da60b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.204576 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.204533 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7bcr\" (UniqueName: \"kubernetes.io/projected/af8db987-1561-4e57-87f0-1573ffc18033-kube-api-access-c7bcr\") pod \"tuned-vq4jw\" (UID: \"af8db987-1561-4e57-87f0-1573ffc18033\") " pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.204922 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.204902 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gtf\" (UniqueName: \"kubernetes.io/projected/2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce-kube-api-access-x2gtf\") pod \"aws-ebs-csi-driver-node-bz7ks\" (UID: \"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.205019 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.205001 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2xw\" (UniqueName: \"kubernetes.io/projected/dc121a5f-2eee-4ca2-a204-fcefee21d9ff-kube-api-access-cm2xw\") pod \"node-ca-rgj2t\" (UID: \"dc121a5f-2eee-4ca2-a204-fcefee21d9ff\") " pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.294834 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294802 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-run-k8s-cni-cncf-io\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.294834 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294838 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02ad25a4-76db-4b91-b212-a701afcd24b8-kubelet-config\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294856 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-var-lib-kubelet\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mfg\" (UniqueName: \"kubernetes.io/projected/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-kube-api-access-h6mfg\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294905 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294916 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-run-k8s-cni-cncf-io\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294927 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-hostroot\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294932 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02ad25a4-76db-4b91-b212-a701afcd24b8-kubelet-config\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294944 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-var-lib-kubelet\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294953 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-run-netns\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294979 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-cnibin\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.294984 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-hostroot\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295000 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00c3d528-0dd0-4c53-99d3-181afc05cf12-cni-binary-copy\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295023 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-conf-dir\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295025 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-run-netns\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295044 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-etc-kubernetes\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295068 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-conf-dir\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295069 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdh7m\" (UniqueName: \"kubernetes.io/projected/00c3d528-0dd0-4c53-99d3-181afc05cf12-kube-api-access-qdh7m\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.295106 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295109 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295124 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-etc-kubernetes\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295138 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02ad25a4-76db-4b91-b212-a701afcd24b8-dbus\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295101 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-cnibin\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295172 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-system-cni-dir\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.295210 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret podName:02ad25a4-76db-4b91-b212-a701afcd24b8 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:16.795170201 +0000 UTC m=+3.127844840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret") pod "global-pull-secret-syncer-vl8zn" (UID: "02ad25a4-76db-4b91-b212-a701afcd24b8") : object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295219 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-system-cni-dir\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295249 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-daemon-config\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295274 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295292 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02ad25a4-76db-4b91-b212-a701afcd24b8-dbus\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295303 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-socket-dir-parent\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295329 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-var-lib-cni-multus\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295356 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-var-lib-cni-bin\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.295378 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295396 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-socket-dir-parent\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295411 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-run-multus-certs\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.295845 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295437 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-var-lib-cni-multus\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.296348 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295442 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-var-lib-cni-bin\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.296348 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295380 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-host-run-multus-certs\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.296348 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.295498 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs podName:dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:16.795479419 +0000 UTC m=+3.128154042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs") pod "network-metrics-daemon-sghnw" (UID: "dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:16.296348 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295560 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-cni-dir\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.296348 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295640 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-cni-dir\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.296348 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295773 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00c3d528-0dd0-4c53-99d3-181afc05cf12-multus-daemon-config\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.296348 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.295796 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00c3d528-0dd0-4c53-99d3-181afc05cf12-cni-binary-copy\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.310232 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.310164 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 13:16:16.310232 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.310189 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 13:16:16.310232 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.310203 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hdjp4 for pod openshift-network-diagnostics/network-check-target-q8rk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:16.310459 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.310279 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4 podName:c0eab916-82d1-4c16-8339-12cabd7e0d6c nodeName:}" failed. No retries permitted until 2026-09-08 13:16:16.810260595 +0000 UTC m=+3.142935228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hdjp4" (UniqueName: "kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4") pod "network-check-target-q8rk6" (UID: "c0eab916-82d1-4c16-8339-12cabd7e0d6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:16.310620 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.310602 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdh7m\" (UniqueName: \"kubernetes.io/projected/00c3d528-0dd0-4c53-99d3-181afc05cf12-kube-api-access-qdh7m\") pod \"multus-zf6qs\" (UID: \"00c3d528-0dd0-4c53-99d3-181afc05cf12\") " pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.312636 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.312609 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mfg\" (UniqueName: \"kubernetes.io/projected/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-kube-api-access-h6mfg\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:16.374814 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.374778 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rgj2t" Sep 08 13:16:16.382166 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.381989 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kvrtd" Sep 08 13:16:16.385343 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:16.385255 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc121a5f_2eee_4ca2_a204_fcefee21d9ff.slice/crio-a69d6bc8c4e2bb4aa5d0ca0547663b80805fc1cb20a9b9218057b548c69a93e2 WatchSource:0}: Error finding container a69d6bc8c4e2bb4aa5d0ca0547663b80805fc1cb20a9b9218057b548c69a93e2: Status 404 returned error can't find the container with id a69d6bc8c4e2bb4aa5d0ca0547663b80805fc1cb20a9b9218057b548c69a93e2 Sep 08 13:16:16.392140 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.392114 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" Sep 08 13:16:16.392884 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:16.392859 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e75a43_fd1d_4b84_b0b6_56cdd64db693.slice/crio-b3bc3fb0a15b8282c320e7d507674fa69a9ad2984b2605c08a5235946451907b WatchSource:0}: Error finding container b3bc3fb0a15b8282c320e7d507674fa69a9ad2984b2605c08a5235946451907b: Status 404 returned error can't find the container with id b3bc3fb0a15b8282c320e7d507674fa69a9ad2984b2605c08a5235946451907b Sep 08 13:16:16.395596 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.395574 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" Sep 08 13:16:16.401976 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.401918 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:16.402604 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:16.402581 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fee6069_2d5a_4cb2_be72_5d0ed4b3e2ce.slice/crio-36ce5900bb44df8e85de2d412da85e72a0689739058cc4a7238f41f4ae979b7a WatchSource:0}: Error finding container 36ce5900bb44df8e85de2d412da85e72a0689739058cc4a7238f41f4ae979b7a: Status 404 returned error can't find the container with id 36ce5900bb44df8e85de2d412da85e72a0689739058cc4a7238f41f4ae979b7a Sep 08 13:16:16.404800 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:16.404775 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8db987_1561_4e57_87f0_1573ffc18033.slice/crio-6409219f7ff24342c87fa7dbc15534fc72e2d5c8b6162633d51932f5bb1eb9e5 WatchSource:0}: Error finding container 6409219f7ff24342c87fa7dbc15534fc72e2d5c8b6162633d51932f5bb1eb9e5: Status 404 returned error can't find the container with id 6409219f7ff24342c87fa7dbc15534fc72e2d5c8b6162633d51932f5bb1eb9e5 Sep 08 13:16:16.406612 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.406556 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hp94q" Sep 08 13:16:16.412070 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.412050 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" Sep 08 13:16:16.412217 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:16.412199 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2047f6b0_59c8_4ec1_bfa7_846d78da60b4.slice/crio-31bf8e073b6a56e58995fc006574c97fa7a91da7fb1dbe198b04cb9eb1749cbb WatchSource:0}: Error finding container 31bf8e073b6a56e58995fc006574c97fa7a91da7fb1dbe198b04cb9eb1749cbb: Status 404 returned error can't find the container with id 31bf8e073b6a56e58995fc006574c97fa7a91da7fb1dbe198b04cb9eb1749cbb Sep 08 13:16:16.416672 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:16.416654 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f87d80_4c14_4f36_bd27_852618164729.slice/crio-81cee2e2c013d51b72672f7f5aa46afad56054daf69b52b60a2f375ceaf5f042 WatchSource:0}: Error finding container 81cee2e2c013d51b72672f7f5aa46afad56054daf69b52b60a2f375ceaf5f042: Status 404 returned error can't find the container with id 81cee2e2c013d51b72672f7f5aa46afad56054daf69b52b60a2f375ceaf5f042 Sep 08 13:16:16.418264 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.418098 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:16.421963 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.421858 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zf6qs" Sep 08 13:16:16.422889 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:16.422862 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311ed079_bea5_4b44_b238_fca763ef6788.slice/crio-1998899d5d8bebd5e89d6d7f4eb1b0e883d6733d8dc153f69e4ca2a2a4367621 WatchSource:0}: Error finding container 1998899d5d8bebd5e89d6d7f4eb1b0e883d6733d8dc153f69e4ca2a2a4367621: Status 404 returned error can't find the container with id 1998899d5d8bebd5e89d6d7f4eb1b0e883d6733d8dc153f69e4ca2a2a4367621 Sep 08 13:16:16.430934 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:16.430911 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944df588_8dba_4a35_80af_b21714f5ca22.slice/crio-263adcea21c1d353b5463fe365ea6a627a6c97ef81a8e8432d31d1027e3ea9c8 WatchSource:0}: Error finding container 263adcea21c1d353b5463fe365ea6a627a6c97ef81a8e8432d31d1027e3ea9c8: Status 404 returned error can't find the container with id 263adcea21c1d353b5463fe365ea6a627a6c97ef81a8e8432d31d1027e3ea9c8 Sep 08 13:16:16.431837 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:16.431815 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c3d528_0dd0_4c53_99d3_181afc05cf12.slice/crio-9a5822f58035dcfa2b19cec25cb88cf5c5a405b5f1de90c31f32ce920dab8023 WatchSource:0}: Error finding container 9a5822f58035dcfa2b19cec25cb88cf5c5a405b5f1de90c31f32ce920dab8023: Status 404 returned error can't find the container with id 9a5822f58035dcfa2b19cec25cb88cf5c5a405b5f1de90c31f32ce920dab8023 Sep 08 13:16:16.798135 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.798094 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:16.798352 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.798176 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:16.798352 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.798211 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:16.798352 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.798265 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:16.798352 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.798280 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret podName:02ad25a4-76db-4b91-b212-a701afcd24b8 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:17.798260534 +0000 UTC m=+4.130935171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret") pod "global-pull-secret-syncer-vl8zn" (UID: "02ad25a4-76db-4b91-b212-a701afcd24b8") : object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:16.798352 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.798299 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs podName:dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:17.798288875 +0000 UTC m=+4.130963507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs") pod "network-metrics-daemon-sghnw" (UID: "dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:16.898919 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:16.898806 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:16.899177 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.899003 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 13:16:16.899177 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.899031 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 13:16:16.899177 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.899044 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hdjp4 for pod openshift-network-diagnostics/network-check-target-q8rk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:16.899177 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:16.899115 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4 podName:c0eab916-82d1-4c16-8339-12cabd7e0d6c nodeName:}" failed. No retries permitted until 2026-09-08 13:16:17.899095512 +0000 UTC m=+4.231770134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdjp4" (UniqueName: "kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4") pod "network-check-target-q8rk6" (UID: "c0eab916-82d1-4c16-8339-12cabd7e0d6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:17.132097 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.132000 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:16 +0000 UTC" deadline="2028-03-03 23:02:09.820526482 +0000 UTC" Sep 08 13:16:17.132097 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.132042 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13017h45m52.688488617s" Sep 08 13:16:17.183426 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.183391 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:17.183643 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:17.183617 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:17.193393 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.193355 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zf6qs" event={"ID":"00c3d528-0dd0-4c53-99d3-181afc05cf12","Type":"ContainerStarted","Data":"9a5822f58035dcfa2b19cec25cb88cf5c5a405b5f1de90c31f32ce920dab8023"} Sep 08 13:16:17.195750 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.195655 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hgc6w" event={"ID":"944df588-8dba-4a35-80af-b21714f5ca22","Type":"ContainerStarted","Data":"263adcea21c1d353b5463fe365ea6a627a6c97ef81a8e8432d31d1027e3ea9c8"} Sep 08 13:16:17.197097 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.197042 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hp94q" event={"ID":"19f87d80-4c14-4f36-bd27-852618164729","Type":"ContainerStarted","Data":"81cee2e2c013d51b72672f7f5aa46afad56054daf69b52b60a2f375ceaf5f042"} Sep 08 13:16:17.198671 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.198585 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" event={"ID":"2047f6b0-59c8-4ec1-bfa7-846d78da60b4","Type":"ContainerStarted","Data":"31bf8e073b6a56e58995fc006574c97fa7a91da7fb1dbe198b04cb9eb1749cbb"} Sep 08 13:16:17.199802 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.199741 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" event={"ID":"af8db987-1561-4e57-87f0-1573ffc18033","Type":"ContainerStarted","Data":"6409219f7ff24342c87fa7dbc15534fc72e2d5c8b6162633d51932f5bb1eb9e5"} Sep 08 13:16:17.201601 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.201543 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" event={"ID":"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce","Type":"ContainerStarted","Data":"36ce5900bb44df8e85de2d412da85e72a0689739058cc4a7238f41f4ae979b7a"} Sep 08 13:16:17.202932 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.202882 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kvrtd" event={"ID":"98e75a43-fd1d-4b84-b0b6-56cdd64db693","Type":"ContainerStarted","Data":"b3bc3fb0a15b8282c320e7d507674fa69a9ad2984b2605c08a5235946451907b"} Sep 08 13:16:17.204344 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.204293 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" event={"ID":"311ed079-bea5-4b44-b238-fca763ef6788","Type":"ContainerStarted","Data":"1998899d5d8bebd5e89d6d7f4eb1b0e883d6733d8dc153f69e4ca2a2a4367621"} Sep 08 13:16:17.205557 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.205508 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rgj2t" event={"ID":"dc121a5f-2eee-4ca2-a204-fcefee21d9ff","Type":"ContainerStarted","Data":"a69d6bc8c4e2bb4aa5d0ca0547663b80805fc1cb20a9b9218057b548c69a93e2"} Sep 08 13:16:17.807523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.807468 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:17.807797 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.807562 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:17.807797 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:17.807695 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:17.807797 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:17.807755 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret podName:02ad25a4-76db-4b91-b212-a701afcd24b8 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:19.807737874 +0000 UTC m=+6.140412504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret") pod "global-pull-secret-syncer-vl8zn" (UID: "02ad25a4-76db-4b91-b212-a701afcd24b8") : object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:17.807994 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:17.807836 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:17.807994 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:17.807875 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs podName:dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:19.807862259 +0000 UTC m=+6.140536890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs") pod "network-metrics-daemon-sghnw" (UID: "dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:17.908900 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:17.908237 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:17.908900 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:17.908459 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 13:16:17.908900 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:17.908482 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 13:16:17.908900 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:17.908494 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hdjp4 for pod openshift-network-diagnostics/network-check-target-q8rk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:17.908900 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:17.908551 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4 podName:c0eab916-82d1-4c16-8339-12cabd7e0d6c nodeName:}" failed. No retries permitted until 2026-09-08 13:16:19.908532952 +0000 UTC m=+6.241207573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdjp4" (UniqueName: "kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4") pod "network-check-target-q8rk6" (UID: "c0eab916-82d1-4c16-8339-12cabd7e0d6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:18.133074 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:18.132936 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:16 +0000 UTC" deadline="2028-05-30 13:08:44.391479296 +0000 UTC" Sep 08 13:16:18.133074 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:18.132989 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15119h52m26.258492216s" Sep 08 13:16:18.183631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:18.183596 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:18.183814 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:18.183609 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:18.183814 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:18.183767 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:18.183968 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:18.183879 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:19.183210 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:19.183133 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:19.183764 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:19.183278 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:19.824818 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:19.824771 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:19.825001 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:19.824850 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:19.825001 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:19.824977 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:19.825128 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:19.825041 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs podName:dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:23.825020735 +0000 UTC m=+10.157695356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs") pod "network-metrics-daemon-sghnw" (UID: "dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:19.825449 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:19.825417 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:19.825583 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:19.825467 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret podName:02ad25a4-76db-4b91-b212-a701afcd24b8 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:23.825452702 +0000 UTC m=+10.158127323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret") pod "global-pull-secret-syncer-vl8zn" (UID: "02ad25a4-76db-4b91-b212-a701afcd24b8") : object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:19.925378 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:19.925341 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:19.925582 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:19.925515 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 13:16:19.925582 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:19.925537 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 13:16:19.925582 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:19.925547 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hdjp4 for pod openshift-network-diagnostics/network-check-target-q8rk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:19.925765 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:19.925608 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4 podName:c0eab916-82d1-4c16-8339-12cabd7e0d6c nodeName:}" failed. No retries permitted until 2026-09-08 13:16:23.925588958 +0000 UTC m=+10.258263579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdjp4" (UniqueName: "kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4") pod "network-check-target-q8rk6" (UID: "c0eab916-82d1-4c16-8339-12cabd7e0d6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:20.183791 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:20.183229 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:20.183791 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:20.183372 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:20.184501 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:20.184358 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:20.184501 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:20.184465 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:21.183379 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:21.183286 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:21.183544 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:21.183440 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:22.183023 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:22.182847 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:22.183023 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:22.182988 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:22.183550 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:22.183434 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:22.183550 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:22.183519 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:23.183113 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:23.183032 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:23.183602 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:23.183176 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:23.860041 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:23.860006 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:23.860207 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:23.860095 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:23.860207 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:23.860164 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:23.860207 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:23.860204 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:23.860353 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:23.860241 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs podName:dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:31.86022005 +0000 UTC m=+18.192894670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs") pod "network-metrics-daemon-sghnw" (UID: "dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:23.860353 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:23.860263 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret podName:02ad25a4-76db-4b91-b212-a701afcd24b8 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:31.860252849 +0000 UTC m=+18.192927465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret") pod "global-pull-secret-syncer-vl8zn" (UID: "02ad25a4-76db-4b91-b212-a701afcd24b8") : object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:23.960677 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:23.960633 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:23.960845 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:23.960831 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 13:16:23.960896 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:23.960856 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 13:16:23.960896 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:23.960868 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hdjp4 for pod openshift-network-diagnostics/network-check-target-q8rk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:23.961004 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:23.960936 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4 podName:c0eab916-82d1-4c16-8339-12cabd7e0d6c nodeName:}" failed. No retries permitted until 2026-09-08 13:16:31.960917103 +0000 UTC m=+18.293591724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdjp4" (UniqueName: "kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4") pod "network-check-target-q8rk6" (UID: "c0eab916-82d1-4c16-8339-12cabd7e0d6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:24.183987 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:24.183906 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:24.184562 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:24.184012 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:24.184562 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:24.184413 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:24.184562 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:24.184520 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:25.183193 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:25.182704 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:25.183193 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:25.182839 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:26.183634 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:26.183598 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:26.184132 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:26.183598 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:26.184132 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:26.183756 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:26.184132 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:26.183831 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:27.182896 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:27.182859 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:27.183068 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:27.182985 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:28.183424 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:28.183381 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:28.183910 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:28.183425 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:28.183910 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:28.183514 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:28.183910 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:28.183661 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:29.183181 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:29.183147 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:29.183335 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:29.183267 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:30.183511 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:30.183419 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:30.183958 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:30.183553 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:30.183958 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:30.183602 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:30.183958 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:30.183751 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:31.182807 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:31.182754 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:31.183093 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:31.182877 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:31.920164 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:31.920127 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:31.920608 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:31.920203 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:31.920608 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:31.920304 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:31.920608 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:31.920370 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs podName:dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:47.92035587 +0000 UTC m=+34.253030499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs") pod "network-metrics-daemon-sghnw" (UID: "dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:31.920608 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:31.920314 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:31.920608 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:31.920446 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret podName:02ad25a4-76db-4b91-b212-a701afcd24b8 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:47.920430476 +0000 UTC m=+34.253105141 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret") pod "global-pull-secret-syncer-vl8zn" (UID: "02ad25a4-76db-4b91-b212-a701afcd24b8") : object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:32.021643 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:32.021607 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:32.021842 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:32.021781 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 13:16:32.021842 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:32.021802 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 13:16:32.021842 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:32.021815 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hdjp4 for pod openshift-network-diagnostics/network-check-target-q8rk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:32.021982 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:32.021883 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4 podName:c0eab916-82d1-4c16-8339-12cabd7e0d6c nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.021865262 +0000 UTC m=+34.354539881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdjp4" (UniqueName: "kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4") pod "network-check-target-q8rk6" (UID: "c0eab916-82d1-4c16-8339-12cabd7e0d6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:32.183304 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:32.183227 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:32.183304 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:32.183246 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:32.183504 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:32.183368 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:32.183558 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:32.183496 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:33.183224 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:33.183182 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:33.183633 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:33.183314 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:34.184875 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.184264 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:34.184875 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:34.184624 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:34.185644 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.185081 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:34.185644 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:34.185176 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:34.246851 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.246704 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" event={"ID":"4d4cca95c4b58a556d25d9ef4b9a311c","Type":"ContainerStarted","Data":"358aca2d93f19a971d39a6aac113cb8aa43dd40e17f7f7d1a6b37946f545248e"} Sep 08 13:16:34.251519 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.251197 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zf6qs" event={"ID":"00c3d528-0dd0-4c53-99d3-181afc05cf12","Type":"ContainerStarted","Data":"7f88f9680b643bcd0de8be6295c86c73920e3aeb6c473da78fd7923306166de4"} Sep 08 13:16:34.255940 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.255646 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hgc6w" event={"ID":"944df588-8dba-4a35-80af-b21714f5ca22","Type":"ContainerStarted","Data":"ce71c76d73c476884fb2220f754f29006aefb7f4bf794e5d0b9695bbe39fdb9d"} Sep 08 13:16:34.263807 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.263580 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" event={"ID":"2047f6b0-59c8-4ec1-bfa7-846d78da60b4","Type":"ContainerStarted","Data":"8847b6f6e354a6ee257a38886318a697dc08a2e37b9971efdf00900175a2617b"} Sep 08 13:16:34.263964 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.263841 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" event={"ID":"2047f6b0-59c8-4ec1-bfa7-846d78da60b4","Type":"ContainerStarted","Data":"766dc9b676fc38e47762f165b1608766896dee356816da08e9f52895d7e25f36"} Sep 08 13:16:34.263964 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.263897 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" event={"ID":"2047f6b0-59c8-4ec1-bfa7-846d78da60b4","Type":"ContainerStarted","Data":"2142bd19b60743b5bc342e106955776f8f577b2d37890df5405cf07fba22dbc4"} Sep 08 13:16:34.264607 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.264452 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" podStartSLOduration=19.26443295 podStartE2EDuration="19.26443295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:34.26409985 +0000 UTC m=+20.596774488" watchObservedRunningTime="2026-09-08 13:16:34.26443295 +0000 UTC m=+20.597107590" Sep 08 13:16:34.267068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.267029 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" event={"ID":"af8db987-1561-4e57-87f0-1573ffc18033","Type":"ContainerStarted","Data":"ea8cb7a0499e574c08d0fd59f6f4f5cdb9a540e5a6682632a7e5f62b59cd48fc"} Sep 08 13:16:34.269869 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.269840 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" event={"ID":"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce","Type":"ContainerStarted","Data":"d1bcf02070da339cffbdcfa2c2936227c6f87a5c6fc3cf67817d1ca537631d30"} Sep 08 13:16:34.273756 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.273649 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" event={"ID":"311ed079-bea5-4b44-b238-fca763ef6788","Type":"ContainerStarted","Data":"0f5d8ab8ebcb54768e4db4765defbe49b249401d077cbc94c1b72e3d69218763"} Sep 08 13:16:34.279012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.278958 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rgj2t" event={"ID":"dc121a5f-2eee-4ca2-a204-fcefee21d9ff","Type":"ContainerStarted","Data":"5120e6cd0ca4536f90f4ec8da34a42ced83bc68db36a70d25f983a42de4b0dab"} Sep 08 13:16:34.282385 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.282333 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hp94q" podStartSLOduration=11.298415682 podStartE2EDuration="20.282313151s" totalImagesPullingTime="8.983897469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:34.281605935 +0000 UTC m=+20.614280574" watchObservedRunningTime="2026-09-08 13:16:34.282313151 +0000 UTC m=+20.614987790" Sep 08 13:16:34.298288 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.298242 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hgc6w" podStartSLOduration=12.07308134 podStartE2EDuration="20.298227639s" totalImagesPullingTime="8.225146299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:34.298072512 +0000 UTC m=+20.630747169" watchObservedRunningTime="2026-09-08 13:16:34.298227639 +0000 UTC m=+20.630902276" Sep 08 13:16:34.337975 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.337890 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zf6qs" podStartSLOduration=2.95567094 podStartE2EDuration="20.33787062s" totalImagesPullingTime="17.38219968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:34.320512511 +0000 UTC m=+20.653187175" watchObservedRunningTime="2026-09-08 13:16:34.33787062 +0000 UTC m=+20.670545277" Sep 08 13:16:34.379585 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.379527 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rgj2t" podStartSLOduration=12.503161363 podStartE2EDuration="20.379506894s" totalImagesPullingTime="7.876345531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:34.356487551 +0000 UTC m=+20.689162200" watchObservedRunningTime="2026-09-08 13:16:34.379506894 +0000 UTC m=+20.712181532" Sep 08 13:16:34.380351 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:34.380150 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vq4jw" podStartSLOduration=6.1980376249999996 podStartE2EDuration="20.380133525s" totalImagesPullingTime="14.1820959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:34.379607621 +0000 UTC m=+20.712282288" watchObservedRunningTime="2026-09-08 13:16:34.380133525 +0000 UTC m=+20.712808165" Sep 08 13:16:35.101552 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.101531 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:35.102059 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.102035 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:35.183034 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.183010 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:35.183151 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:35.183130 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:35.188137 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.188116 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 13:16:35.282062 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.282026 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kvrtd" event={"ID":"98e75a43-fd1d-4b84-b0b6-56cdd64db693","Type":"ContainerStarted","Data":"d73764b5ff25e07bfe8c011f5736cd049f19abbe162f19462c50dc8cd181aa0b"} Sep 08 13:16:35.283789 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.283763 2554 generic.go:356] "Generic (PLEG): container finished" podID="8629079a46d2d7de6ec5fcd3cb9f06dc" containerID="18061a72cde29c69921134114b8329e52dba500f594629eeb78f4fbcf497b249" exitCode=0 Sep 08 13:16:35.283880 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.283836 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" event={"ID":"8629079a46d2d7de6ec5fcd3cb9f06dc","Type":"ContainerDied","Data":"18061a72cde29c69921134114b8329e52dba500f594629eeb78f4fbcf497b249"} Sep 08 13:16:35.283880 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.283877 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" event={"ID":"8629079a46d2d7de6ec5fcd3cb9f06dc","Type":"ContainerStarted","Data":"26c8ea40d1b3ab4dbc9dfc24d21b7dcec33cfea9ae0126f798f60710286dabb9"} Sep 08 13:16:35.285247 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.285220 2554 generic.go:356] "Generic (PLEG): container finished" podID="311ed079-bea5-4b44-b238-fca763ef6788" containerID="0f5d8ab8ebcb54768e4db4765defbe49b249401d077cbc94c1b72e3d69218763" exitCode=0 Sep 08 13:16:35.285337 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.285307 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" event={"ID":"311ed079-bea5-4b44-b238-fca763ef6788","Type":"ContainerDied","Data":"0f5d8ab8ebcb54768e4db4765defbe49b249401d077cbc94c1b72e3d69218763"} Sep 08 13:16:35.286771 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.286751 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hp94q" event={"ID":"19f87d80-4c14-4f36-bd27-852618164729","Type":"ContainerStarted","Data":"76060f1bccf868fe33c6a91054659ae40a068a4cb15f7256d971637551e773ce"} Sep 08 13:16:35.289995 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.289974 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" event={"ID":"2047f6b0-59c8-4ec1-bfa7-846d78da60b4","Type":"ContainerStarted","Data":"b6fd36b65c2076dd4f1401b65f9ef45d4fccc4b30741d14885a30b2079b76879"} Sep 08 13:16:35.290101 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.290045 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" event={"ID":"2047f6b0-59c8-4ec1-bfa7-846d78da60b4","Type":"ContainerStarted","Data":"543eb5664e788617ae9247300317097222f18b326bb779f088bcf7bd6578dd9d"} Sep 08 13:16:35.290160 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.290098 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" event={"ID":"2047f6b0-59c8-4ec1-bfa7-846d78da60b4","Type":"ContainerStarted","Data":"d382197e5a0ce1d0b28c0ac61990da8615ef229fe19cfcbb0584ed5a20471014"} Sep 08 13:16:35.291832 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.291807 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" event={"ID":"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce","Type":"ContainerStarted","Data":"edb1f648fd44b4697d4eca7c41393998ece3cf9c8569b6494a2a23a68eb65644"} Sep 08 13:16:35.292332 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.292315 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:35.292705 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.292673 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hgc6w" Sep 08 13:16:35.308012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.307975 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kvrtd" podStartSLOduration=3.886321381 podStartE2EDuration="21.307964315s" totalImagesPullingTime="17.421642934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:35.307834965 +0000 UTC m=+21.640509603" watchObservedRunningTime="2026-09-08 13:16:35.307964315 +0000 UTC m=+21.640638953" Sep 08 13:16:35.336284 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:35.336242 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" podStartSLOduration=20.336229692 podStartE2EDuration="20.336229692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:35.33609256 +0000 UTC m=+21.668767197" watchObservedRunningTime="2026-09-08 13:16:35.336229692 +0000 UTC m=+21.668904329" Sep 08 13:16:35.622300 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:35.622254 2554 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce: reading manifest sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce" Sep 08 13:16:35.622458 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:35.622368 2554 kuberuntime_manager.go:1459] "Unhandled Error" err="init container cni-plugins start failed in pod multus-additional-cni-plugins-v6zgx_openshift-multus(311ed079-bea5-4b44-b238-fca763ef6788): ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce: reading manifest sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image" logger="UnhandledError" Sep 08 13:16:35.622458 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:35.622404 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cni-plugins\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce: reading manifest sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\"" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" podUID="311ed079-bea5-4b44-b238-fca763ef6788" Sep 08 13:16:36.153703 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:36.153369 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:16:35.188132673Z","UUID":"281b3445-7a03-4221-8125-b91a8ddd23ee","Handler":null,"Name":"","Endpoint":""} Sep 08 13:16:36.154907 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:36.154888 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 13:16:36.155033 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:36.154921 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 13:16:36.183247 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:36.183216 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:36.183372 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:36.183313 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:36.183433 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:36.183410 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:36.183533 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:36.183512 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:36.295925 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:36.295800 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" event={"ID":"2fee6069-2d5a-4cb2-be72-5d0ed4b3e2ce","Type":"ContainerStarted","Data":"b0f9da0042d69f1bfb57409b9988083a18aa526f28fc9e6fbd0390b74fb71ba6"} Sep 08 13:16:36.297033 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:36.297004 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cni-plugins\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce: reading manifest sha256:1c6b55e6f6dc3f07ab2626923169873734c7376bf8660d673a127e97047ed6ce in quay.io/openshift-release-dev/ocp-v4.0-art-dev: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\"" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" podUID="311ed079-bea5-4b44-b238-fca763ef6788" Sep 08 13:16:36.337771 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:36.337720 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bz7ks" podStartSLOduration=7.316091528 podStartE2EDuration="22.337707098s" totalImagesPullingTime="15.02161557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:36.337094243 +0000 UTC m=+22.669768880" watchObservedRunningTime="2026-09-08 13:16:36.337707098 +0000 UTC m=+22.670381767" Sep 08 13:16:37.182987 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:37.182952 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:37.183141 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:37.183050 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:37.300583 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:37.300547 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" event={"ID":"2047f6b0-59c8-4ec1-bfa7-846d78da60b4","Type":"ContainerStarted","Data":"1222efc24d2d031d9f477944cf94a391c9c2bd0cf8d7fce9608ca2743ccfa258"} Sep 08 13:16:38.183042 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:38.183003 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:38.183330 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:38.183123 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:38.183330 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:38.183183 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:38.183330 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:38.183291 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:39.183128 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:39.183094 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:39.183482 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:39.183204 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:39.308638 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:39.308596 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" event={"ID":"2047f6b0-59c8-4ec1-bfa7-846d78da60b4","Type":"ContainerStarted","Data":"940e1ab63c7ac5d416045e94371c55fd67edc3b443ea4cca7ed1df664a678331"} Sep 08 13:16:39.308947 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:39.308919 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:39.309070 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:39.308996 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:39.324276 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:39.324245 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:39.336877 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:39.336841 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" podStartSLOduration=7.833615753 podStartE2EDuration="25.33682891s" totalImagesPullingTime="17.503213157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:39.336289188 +0000 UTC m=+25.668963825" watchObservedRunningTime="2026-09-08 13:16:39.33682891 +0000 UTC m=+25.669503548" Sep 08 13:16:40.183857 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.183662 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:40.184286 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.183662 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:40.184286 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:40.183945 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:40.184286 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:40.184003 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:40.310735 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.310709 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:40.325232 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.325198 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:16:40.496872 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.496790 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sghnw"] Sep 08 13:16:40.497003 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.496929 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:40.497057 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:40.497024 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:40.499779 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.499753 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vl8zn"] Sep 08 13:16:40.499904 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.499857 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:40.499976 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:40.499952 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:40.502668 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.502646 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8rk6"] Sep 08 13:16:40.502777 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:40.502763 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:40.502856 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:40.502840 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:42.183506 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:42.183475 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:42.183966 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:42.183479 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:42.183966 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:42.183598 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:42.183966 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:42.183481 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:42.183966 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:42.183661 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:42.183966 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:42.183742 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:44.186040 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:44.186003 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:44.186040 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:44.186042 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:44.186633 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:44.186108 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:44.186633 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:44.186117 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:44.186633 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:44.186245 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:44.186633 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:44.186332 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:46.182921 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.182881 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:46.183509 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.182881 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:46.183509 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:46.183021 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:16:46.183509 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.182884 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:46.183509 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:46.183101 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8rk6" podUID="c0eab916-82d1-4c16-8339-12cabd7e0d6c" Sep 08 13:16:46.183509 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:46.183138 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl8zn" podUID="02ad25a4-76db-4b91-b212-a701afcd24b8" Sep 08 13:16:46.425966 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.425934 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeReady" Sep 08 13:16:46.426150 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.426080 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 13:16:46.462107 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.462032 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68f6f5b69b-h9jwj"] Sep 08 13:16:46.464037 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.464022 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.466531 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.466510 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 13:16:46.466649 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.466607 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 13:16:46.466649 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.466615 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s2n2x\"" Sep 08 13:16:46.466795 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.466640 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 13:16:46.472923 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.472901 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 13:16:46.477537 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.477515 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f6f5b69b-h9jwj"] Sep 08 13:16:46.481417 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.481393 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w9xj9"] Sep 08 13:16:46.483375 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.483358 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z69vl"] Sep 08 13:16:46.483497 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.483483 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.485059 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.485043 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z69vl" Sep 08 13:16:46.486136 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.486108 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6zj7\"" Sep 08 13:16:46.486235 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.486161 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 13:16:46.486235 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.486187 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 13:16:46.487118 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.487103 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 13:16:46.487533 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.487517 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 13:16:46.487858 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.487842 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-89flp\"" Sep 08 13:16:46.487934 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.487921 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 13:16:46.499970 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.499945 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z69vl"] Sep 08 13:16:46.500567 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.500535 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w9xj9"] Sep 08 13:16:46.619004 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.618960 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-installation-pull-secrets\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.619200 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619017 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-image-registry-private-configuration\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.619200 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619044 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.619200 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619067 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41ab8e0e-1c37-43e4-a96e-f602e8995304-ca-trust-extracted\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.619200 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619123 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-bound-sa-token\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.619200 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619145 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjrs\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-kube-api-access-nvjrs\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.619406 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619192 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-certificates\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.619406 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619226 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-trusted-ca\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.619406 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619265 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19ee7d05-e185-443a-ba8e-deb5446dd2ab-config-volume\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.619406 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619281 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.619406 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619295 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ee7d05-e185-443a-ba8e-deb5446dd2ab-tmp-dir\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.619406 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619312 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c357275f-3687-4075-bffd-b8acd4f279c4-cert\") pod \"ingress-canary-z69vl\" (UID: \"c357275f-3687-4075-bffd-b8acd4f279c4\") " pod="openshift-ingress-canary/ingress-canary-z69vl" Sep 08 13:16:46.619406 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619347 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njdtn\" (UniqueName: \"kubernetes.io/projected/c357275f-3687-4075-bffd-b8acd4f279c4-kube-api-access-njdtn\") pod \"ingress-canary-z69vl\" (UID: \"c357275f-3687-4075-bffd-b8acd4f279c4\") " pod="openshift-ingress-canary/ingress-canary-z69vl" Sep 08 13:16:46.619614 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.619408 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8p98\" (UniqueName: \"kubernetes.io/projected/19ee7d05-e185-443a-ba8e-deb5446dd2ab-kube-api-access-s8p98\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.632363 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.632329 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hgc6w_944df588-8dba-4a35-80af-b21714f5ca22/konnectivity-agent/0.log" limit=1024 Sep 08 13:16:46.719914 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.719826 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41ab8e0e-1c37-43e4-a96e-f602e8995304-ca-trust-extracted\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.719914 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.719886 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-bound-sa-token\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.719914 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.719910 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjrs\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-kube-api-access-nvjrs\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.720232 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.719934 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-certificates\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.720232 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.719952 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-trusted-ca\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.720232 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720092 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19ee7d05-e185-443a-ba8e-deb5446dd2ab-config-volume\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.720232 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720138 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.720232 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720169 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ee7d05-e185-443a-ba8e-deb5446dd2ab-tmp-dir\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.720232 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720192 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c357275f-3687-4075-bffd-b8acd4f279c4-cert\") pod \"ingress-canary-z69vl\" (UID: \"c357275f-3687-4075-bffd-b8acd4f279c4\") " pod="openshift-ingress-canary/ingress-canary-z69vl" Sep 08 13:16:46.720232 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720219 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njdtn\" (UniqueName: \"kubernetes.io/projected/c357275f-3687-4075-bffd-b8acd4f279c4-kube-api-access-njdtn\") pod \"ingress-canary-z69vl\" (UID: \"c357275f-3687-4075-bffd-b8acd4f279c4\") " pod="openshift-ingress-canary/ingress-canary-z69vl" Sep 08 13:16:46.720579 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720250 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8p98\" (UniqueName: \"kubernetes.io/projected/19ee7d05-e185-443a-ba8e-deb5446dd2ab-kube-api-access-s8p98\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.720579 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:46.720282 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 13:16:46.720579 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720292 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41ab8e0e-1c37-43e4-a96e-f602e8995304-ca-trust-extracted\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.720579 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:46.720313 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f6f5b69b-h9jwj: secret "image-registry-tls" not found Sep 08 13:16:46.720579 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:46.720371 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls podName:41ab8e0e-1c37-43e4-a96e-f602e8995304 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:47.220352865 +0000 UTC m=+33.553027488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls") pod "image-registry-68f6f5b69b-h9jwj" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304") : secret "image-registry-tls" not found Sep 08 13:16:46.720579 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720310 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-installation-pull-secrets\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.720579 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720449 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-image-registry-private-configuration\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.720579 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720498 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.720991 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:46.720607 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 13:16:46.720991 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:46.720666 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls podName:19ee7d05-e185-443a-ba8e-deb5446dd2ab nodeName:}" failed. No retries permitted until 2026-09-08 13:16:47.220652569 +0000 UTC m=+33.553327184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls") pod "dns-default-w9xj9" (UID: "19ee7d05-e185-443a-ba8e-deb5446dd2ab") : secret "dns-default-metrics-tls" not found Sep 08 13:16:46.720991 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720796 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-certificates\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.720991 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.720878 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19ee7d05-e185-443a-ba8e-deb5446dd2ab-config-volume\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.721357 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.721306 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hgc6w_944df588-8dba-4a35-80af-b21714f5ca22/konnectivity-agent/0.log" limit=1024 Sep 08 13:16:46.721512 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.721389 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-trusted-ca\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.721512 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.721434 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ee7d05-e185-443a-ba8e-deb5446dd2ab-tmp-dir\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.724139 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.724115 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-installation-pull-secrets\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.724222 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.724175 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-image-registry-private-configuration\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.724264 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.724251 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c357275f-3687-4075-bffd-b8acd4f279c4-cert\") pod \"ingress-canary-z69vl\" (UID: \"c357275f-3687-4075-bffd-b8acd4f279c4\") " pod="openshift-ingress-canary/ingress-canary-z69vl" Sep 08 13:16:46.731082 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.731061 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjrs\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-kube-api-access-nvjrs\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.731156 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.731090 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8p98\" (UniqueName: \"kubernetes.io/projected/19ee7d05-e185-443a-ba8e-deb5446dd2ab-kube-api-access-s8p98\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:46.731391 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.731371 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njdtn\" (UniqueName: \"kubernetes.io/projected/c357275f-3687-4075-bffd-b8acd4f279c4-kube-api-access-njdtn\") pod \"ingress-canary-z69vl\" (UID: \"c357275f-3687-4075-bffd-b8acd4f279c4\") " pod="openshift-ingress-canary/ingress-canary-z69vl" Sep 08 13:16:46.731513 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.731494 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-bound-sa-token\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:46.800831 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:46.800799 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z69vl" Sep 08 13:16:47.085155 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:47.084982 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z69vl"] Sep 08 13:16:47.088398 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:47.088378 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc357275f_3687_4075_bffd_b8acd4f279c4.slice/crio-eec7058e6cb510a7741e5e157731e3cfaf192e23e45d401f07ee0dfbd2517386 WatchSource:0}: Error finding container eec7058e6cb510a7741e5e157731e3cfaf192e23e45d401f07ee0dfbd2517386: Status 404 returned error can't find the container with id eec7058e6cb510a7741e5e157731e3cfaf192e23e45d401f07ee0dfbd2517386 Sep 08 13:16:47.224523 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:47.224488 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:47.225063 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:47.224545 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:47.225063 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:47.224643 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 13:16:47.225063 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:47.224654 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f6f5b69b-h9jwj: secret "image-registry-tls" not found Sep 08 13:16:47.225063 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:47.224653 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 13:16:47.225063 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:47.224732 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls podName:41ab8e0e-1c37-43e4-a96e-f602e8995304 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.224718174 +0000 UTC m=+34.557392795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls") pod "image-registry-68f6f5b69b-h9jwj" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304") : secret "image-registry-tls" not found Sep 08 13:16:47.225063 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:47.224754 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls podName:19ee7d05-e185-443a-ba8e-deb5446dd2ab nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.22473768 +0000 UTC m=+34.557412297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls") pod "dns-default-w9xj9" (UID: "19ee7d05-e185-443a-ba8e-deb5446dd2ab") : secret "dns-default-metrics-tls" not found Sep 08 13:16:47.327872 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:47.327836 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z69vl" event={"ID":"c357275f-3687-4075-bffd-b8acd4f279c4","Type":"ContainerStarted","Data":"eec7058e6cb510a7741e5e157731e3cfaf192e23e45d401f07ee0dfbd2517386"} Sep 08 13:16:47.929867 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:47.929828 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:47.930083 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:47.929916 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:47.930083 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:47.929982 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:47.930083 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:47.930016 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:47.930083 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:47.930050 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs podName:dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:19.930035448 +0000 UTC m=+66.262710064 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs") pod "network-metrics-daemon-sghnw" (UID: "dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 13:16:47.930083 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:47.930064 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret podName:02ad25a4-76db-4b91-b212-a701afcd24b8 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:19.930058136 +0000 UTC m=+66.262732751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret") pod "global-pull-secret-syncer-vl8zn" (UID: "02ad25a4-76db-4b91-b212-a701afcd24b8") : object "kube-system"/"original-pull-secret" not registered Sep 08 13:16:48.030248 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.030208 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:48.030399 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:48.030375 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 13:16:48.030399 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:48.030395 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 13:16:48.030484 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:48.030407 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hdjp4 for pod openshift-network-diagnostics/network-check-target-q8rk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:48.030484 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:48.030455 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4 podName:c0eab916-82d1-4c16-8339-12cabd7e0d6c nodeName:}" failed. No retries permitted until 2026-09-08 13:17:20.030443202 +0000 UTC m=+66.363117818 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdjp4" (UniqueName: "kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4") pod "network-check-target-q8rk6" (UID: "c0eab916-82d1-4c16-8339-12cabd7e0d6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 13:16:48.183459 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.183384 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:16:48.183631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.183387 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:16:48.183631 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.183387 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:16:48.189941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.189918 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 13:16:48.190615 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.190598 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7tdz6\"" Sep 08 13:16:48.190736 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.190620 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lvcfq\"" Sep 08 13:16:48.190736 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.190620 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 13:16:48.190736 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.190663 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 13:16:48.190736 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.190705 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 13:16:48.231912 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.231884 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:48.232285 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:48.231940 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:48.232285 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:48.232039 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 13:16:48.232285 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:48.232039 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 13:16:48.232285 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:48.232121 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls podName:19ee7d05-e185-443a-ba8e-deb5446dd2ab nodeName:}" failed. No retries permitted until 2026-09-08 13:16:50.232106802 +0000 UTC m=+36.564781419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls") pod "dns-default-w9xj9" (UID: "19ee7d05-e185-443a-ba8e-deb5446dd2ab") : secret "dns-default-metrics-tls" not found Sep 08 13:16:48.232285 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:48.232049 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f6f5b69b-h9jwj: secret "image-registry-tls" not found Sep 08 13:16:48.232285 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:48.232201 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls podName:41ab8e0e-1c37-43e4-a96e-f602e8995304 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:50.232180742 +0000 UTC m=+36.564855373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls") pod "image-registry-68f6f5b69b-h9jwj" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304") : secret "image-registry-tls" not found Sep 08 13:16:49.108673 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:49.108634 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hgc6w_944df588-8dba-4a35-80af-b21714f5ca22/konnectivity-agent/0.log" limit=1024 Sep 08 13:16:49.206361 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:49.206307 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hgc6w_944df588-8dba-4a35-80af-b21714f5ca22/konnectivity-agent/0.log" limit=1024 Sep 08 13:16:50.251071 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:50.251035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:50.251478 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:50.251097 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:50.251478 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:50.251187 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 13:16:50.251478 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:50.251198 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f6f5b69b-h9jwj: secret "image-registry-tls" not found Sep 08 13:16:50.251478 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:50.251202 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 13:16:50.251478 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:50.251249 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls podName:41ab8e0e-1c37-43e4-a96e-f602e8995304 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:54.251234972 +0000 UTC m=+40.583909589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls") pod "image-registry-68f6f5b69b-h9jwj" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304") : secret "image-registry-tls" not found Sep 08 13:16:50.251478 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:50.251261 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls podName:19ee7d05-e185-443a-ba8e-deb5446dd2ab nodeName:}" failed. No retries permitted until 2026-09-08 13:16:54.251255802 +0000 UTC m=+40.583930417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls") pod "dns-default-w9xj9" (UID: "19ee7d05-e185-443a-ba8e-deb5446dd2ab") : secret "dns-default-metrics-tls" not found Sep 08 13:16:51.055814 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.055618 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv"] Sep 08 13:16:51.081660 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.081617 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf"] Sep 08 13:16:51.081861 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.081794 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" Sep 08 13:16:51.084930 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.084910 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 08 13:16:51.085071 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.084908 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 08 13:16:51.087104 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.087081 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hwpzc\"" Sep 08 13:16:51.087221 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.087198 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 08 13:16:51.087274 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.087220 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 08 13:16:51.097713 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.097660 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752"] Sep 08 13:16:51.097830 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.097795 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.100957 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.100933 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 08 13:16:51.100957 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.100949 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 08 13:16:51.101211 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.101192 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 08 13:16:51.101713 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.101671 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 08 13:16:51.119790 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.119765 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv"] Sep 08 13:16:51.119910 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.119803 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf"] Sep 08 13:16:51.119910 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.119826 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752"] Sep 08 13:16:51.119910 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.119898 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.122482 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.122463 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 08 13:16:51.159367 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.159329 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c0cb302-4ba3-417d-a7f0-c8d2c3766282-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv\" (UID: \"7c0cb302-4ba3-417d-a7f0-c8d2c3766282\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" Sep 08 13:16:51.159522 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.159368 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtxrb\" (UniqueName: \"kubernetes.io/projected/7c0cb302-4ba3-417d-a7f0-c8d2c3766282-kube-api-access-mtxrb\") pod \"managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv\" (UID: \"7c0cb302-4ba3-417d-a7f0-c8d2c3766282\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" Sep 08 13:16:51.259959 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.259923 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f28adea-856b-4569-a468-44b8a0be3505-tmp\") pod \"klusterlet-addon-workmgr-7966b9f4b6-kh752\" (UID: \"1f28adea-856b-4569-a468-44b8a0be3505\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.259959 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.259959 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1f28adea-856b-4569-a468-44b8a0be3505-klusterlet-config\") pod \"klusterlet-addon-workmgr-7966b9f4b6-kh752\" (UID: \"1f28adea-856b-4569-a468-44b8a0be3505\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.260447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.259978 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-ca\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.260447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.260018 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/427a1265-e971-45c0-ab4b-3d14ec110ac8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.260447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.260052 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f722w\" (UniqueName: \"kubernetes.io/projected/1f28adea-856b-4569-a468-44b8a0be3505-kube-api-access-f722w\") pod \"klusterlet-addon-workmgr-7966b9f4b6-kh752\" (UID: \"1f28adea-856b-4569-a468-44b8a0be3505\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.260447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.260077 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c0cb302-4ba3-417d-a7f0-c8d2c3766282-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv\" (UID: \"7c0cb302-4ba3-417d-a7f0-c8d2c3766282\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" Sep 08 13:16:51.260447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.260094 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtxrb\" (UniqueName: \"kubernetes.io/projected/7c0cb302-4ba3-417d-a7f0-c8d2c3766282-kube-api-access-mtxrb\") pod \"managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv\" (UID: \"7c0cb302-4ba3-417d-a7f0-c8d2c3766282\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" Sep 08 13:16:51.260447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.260220 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-hub\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.260447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.260283 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.260447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.260334 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62kk\" (UniqueName: \"kubernetes.io/projected/427a1265-e971-45c0-ab4b-3d14ec110ac8-kube-api-access-s62kk\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.260447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.260426 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.264158 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.264138 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c0cb302-4ba3-417d-a7f0-c8d2c3766282-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv\" (UID: \"7c0cb302-4ba3-417d-a7f0-c8d2c3766282\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" Sep 08 13:16:51.269610 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.269588 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtxrb\" (UniqueName: \"kubernetes.io/projected/7c0cb302-4ba3-417d-a7f0-c8d2c3766282-kube-api-access-mtxrb\") pod \"managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv\" (UID: \"7c0cb302-4ba3-417d-a7f0-c8d2c3766282\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" Sep 08 13:16:51.338620 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.338536 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z69vl" event={"ID":"c357275f-3687-4075-bffd-b8acd4f279c4","Type":"ContainerStarted","Data":"fd017b7cc3a22855cfc3c3611f2b27a5024d452a9748dd55a76ccf4e48b7b748"} Sep 08 13:16:51.361022 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.360990 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.361216 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.361044 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f28adea-856b-4569-a468-44b8a0be3505-tmp\") pod \"klusterlet-addon-workmgr-7966b9f4b6-kh752\" (UID: \"1f28adea-856b-4569-a468-44b8a0be3505\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.361216 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.361149 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1f28adea-856b-4569-a468-44b8a0be3505-klusterlet-config\") pod \"klusterlet-addon-workmgr-7966b9f4b6-kh752\" (UID: \"1f28adea-856b-4569-a468-44b8a0be3505\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.361216 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.361177 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-ca\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.361216 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.361198 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/427a1265-e971-45c0-ab4b-3d14ec110ac8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.361414 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.361241 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f722w\" (UniqueName: \"kubernetes.io/projected/1f28adea-856b-4569-a468-44b8a0be3505-kube-api-access-f722w\") pod \"klusterlet-addon-workmgr-7966b9f4b6-kh752\" (UID: \"1f28adea-856b-4569-a468-44b8a0be3505\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.361414 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.361305 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-hub\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.361581 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.361502 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.361581 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.361513 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f28adea-856b-4569-a468-44b8a0be3505-tmp\") pod \"klusterlet-addon-workmgr-7966b9f4b6-kh752\" (UID: \"1f28adea-856b-4569-a468-44b8a0be3505\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.361581 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.361573 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62kk\" (UniqueName: \"kubernetes.io/projected/427a1265-e971-45c0-ab4b-3d14ec110ac8-kube-api-access-s62kk\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.362046 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.362008 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/427a1265-e971-45c0-ab4b-3d14ec110ac8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.364363 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.364338 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-ca\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.364530 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.364487 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-hub\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.364530 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.364500 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.364812 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.364792 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/427a1265-e971-45c0-ab4b-3d14ec110ac8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.366980 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.366951 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1f28adea-856b-4569-a468-44b8a0be3505-klusterlet-config\") pod \"klusterlet-addon-workmgr-7966b9f4b6-kh752\" (UID: \"1f28adea-856b-4569-a468-44b8a0be3505\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.374943 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.374906 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f722w\" (UniqueName: \"kubernetes.io/projected/1f28adea-856b-4569-a468-44b8a0be3505-kube-api-access-f722w\") pod \"klusterlet-addon-workmgr-7966b9f4b6-kh752\" (UID: \"1f28adea-856b-4569-a468-44b8a0be3505\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.375161 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.375134 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62kk\" (UniqueName: \"kubernetes.io/projected/427a1265-e971-45c0-ab4b-3d14ec110ac8-kube-api-access-s62kk\") pod \"cluster-proxy-proxy-agent-7dbf7b898c-m5xpf\" (UID: \"427a1265-e971-45c0-ab4b-3d14ec110ac8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.398832 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.398798 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" Sep 08 13:16:51.406977 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.406943 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:16:51.437913 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.437886 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:16:51.703190 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.703078 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z69vl" podStartSLOduration=2.39206983 podStartE2EDuration="5.703055656s" totalImagesPullingTime="3.310985826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:51.358589913 +0000 UTC m=+37.691264553" watchObservedRunningTime="2026-09-08 13:16:51.703055656 +0000 UTC m=+38.035730292" Sep 08 13:16:51.703369 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.703268 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv"] Sep 08 13:16:51.706809 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:51.706780 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c0cb302_4ba3_417d_a7f0_c8d2c3766282.slice/crio-7924dd3621a3acc3fc75a597ac6954fe3e79c033999306bd70f37097c57f79c3 WatchSource:0}: Error finding container 7924dd3621a3acc3fc75a597ac6954fe3e79c033999306bd70f37097c57f79c3: Status 404 returned error can't find the container with id 7924dd3621a3acc3fc75a597ac6954fe3e79c033999306bd70f37097c57f79c3 Sep 08 13:16:51.738646 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.738547 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf"] Sep 08 13:16:51.741486 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:51.741437 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod427a1265_e971_45c0_ab4b_3d14ec110ac8.slice/crio-f8bac702248889dfe04a6ec082be5358d39226a91c26305742cf575d49956399 WatchSource:0}: Error finding container f8bac702248889dfe04a6ec082be5358d39226a91c26305742cf575d49956399: Status 404 returned error can't find the container with id f8bac702248889dfe04a6ec082be5358d39226a91c26305742cf575d49956399 Sep 08 13:16:51.759608 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:51.759582 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752"] Sep 08 13:16:51.760560 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:16:51.760518 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f28adea_856b_4569_a468_44b8a0be3505.slice/crio-c929577275ac1dbc155d3c7f7fb239706bb7fe5ca8eb230c2a9abe478970132d WatchSource:0}: Error finding container c929577275ac1dbc155d3c7f7fb239706bb7fe5ca8eb230c2a9abe478970132d: Status 404 returned error can't find the container with id c929577275ac1dbc155d3c7f7fb239706bb7fe5ca8eb230c2a9abe478970132d Sep 08 13:16:52.341588 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:52.341547 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" event={"ID":"7c0cb302-4ba3-417d-a7f0-c8d2c3766282","Type":"ContainerStarted","Data":"7924dd3621a3acc3fc75a597ac6954fe3e79c033999306bd70f37097c57f79c3"} Sep 08 13:16:52.342692 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:52.342655 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" event={"ID":"427a1265-e971-45c0-ab4b-3d14ec110ac8","Type":"ContainerStarted","Data":"f8bac702248889dfe04a6ec082be5358d39226a91c26305742cf575d49956399"} Sep 08 13:16:52.343734 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:52.343704 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" event={"ID":"1f28adea-856b-4569-a468-44b8a0be3505","Type":"ContainerStarted","Data":"c929577275ac1dbc155d3c7f7fb239706bb7fe5ca8eb230c2a9abe478970132d"} Sep 08 13:16:54.288333 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:54.288279 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:16:54.288805 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:54.288442 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 13:16:54.288805 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:54.288504 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls podName:19ee7d05-e185-443a-ba8e-deb5446dd2ab nodeName:}" failed. No retries permitted until 2026-09-08 13:17:02.288485185 +0000 UTC m=+48.621159818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls") pod "dns-default-w9xj9" (UID: "19ee7d05-e185-443a-ba8e-deb5446dd2ab") : secret "dns-default-metrics-tls" not found Sep 08 13:16:54.288924 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:54.288899 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:16:54.289059 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:54.289044 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 13:16:54.289131 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:54.289060 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f6f5b69b-h9jwj: secret "image-registry-tls" not found Sep 08 13:16:54.289131 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:16:54.289108 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls podName:41ab8e0e-1c37-43e4-a96e-f602e8995304 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:02.289093499 +0000 UTC m=+48.621768119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls") pod "image-registry-68f6f5b69b-h9jwj" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304") : secret "image-registry-tls" not found Sep 08 13:16:59.367087 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:59.367042 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" event={"ID":"7c0cb302-4ba3-417d-a7f0-c8d2c3766282","Type":"ContainerStarted","Data":"fa126ed629166eb52b4259d34b220347903b08e2681105bfdd28d66d64e891d2"} Sep 08 13:16:59.370117 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:59.370090 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" event={"ID":"427a1265-e971-45c0-ab4b-3d14ec110ac8","Type":"ContainerStarted","Data":"aaecc6987ddc90f990c0869860f9397c18fba9020d03a19491ecc1ff902390ea"} Sep 08 13:16:59.370233 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:59.370132 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" event={"ID":"427a1265-e971-45c0-ab4b-3d14ec110ac8","Type":"ContainerStarted","Data":"3e78fc60fc53d84e6723aef61894fbef80e84323e0ebc8023d9ad80cfeced5db"} Sep 08 13:16:59.370233 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:59.370146 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" event={"ID":"427a1265-e971-45c0-ab4b-3d14ec110ac8","Type":"ContainerStarted","Data":"cd295f3c5e3a1c47209789836ba93d569a62cd6674f34c26cf3ba21e5e3df3b7"} Sep 08 13:16:59.372512 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:59.372485 2554 generic.go:356] "Generic (PLEG): container finished" podID="311ed079-bea5-4b44-b238-fca763ef6788" containerID="eaab622ddb3a78c9c233229f0dcd8fa2f8969cbe53ca30390befbf235e8c25e2" exitCode=0 Sep 08 13:16:59.372512 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:59.372504 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" event={"ID":"311ed079-bea5-4b44-b238-fca763ef6788","Type":"ContainerDied","Data":"eaab622ddb3a78c9c233229f0dcd8fa2f8969cbe53ca30390befbf235e8c25e2"} Sep 08 13:16:59.384886 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:59.384834 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" podStartSLOduration=1.6472304389999999 podStartE2EDuration="8.384814894s" totalImagesPullingTime="6.737584455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:59.383852538 +0000 UTC m=+45.716527174" watchObservedRunningTime="2026-09-08 13:16:59.384814894 +0000 UTC m=+45.717489533" Sep 08 13:16:59.423479 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:16:59.423417 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" podStartSLOduration=1.700719694 podStartE2EDuration="8.423397151s" totalImagesPullingTime="6.722677457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:59.422907831 +0000 UTC m=+45.755582468" watchObservedRunningTime="2026-09-08 13:16:59.423397151 +0000 UTC m=+45.756071803" Sep 08 13:17:02.358661 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:02.358632 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:17:02.359043 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:02.358709 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:17:02.359043 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:02.358790 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 13:17:02.359043 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:02.358822 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 13:17:02.359043 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:02.358834 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f6f5b69b-h9jwj: secret "image-registry-tls" not found Sep 08 13:17:02.359043 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:02.358851 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls podName:19ee7d05-e185-443a-ba8e-deb5446dd2ab nodeName:}" failed. No retries permitted until 2026-09-08 13:17:18.358836008 +0000 UTC m=+64.691510624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls") pod "dns-default-w9xj9" (UID: "19ee7d05-e185-443a-ba8e-deb5446dd2ab") : secret "dns-default-metrics-tls" not found Sep 08 13:17:02.359043 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:02.358869 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls podName:41ab8e0e-1c37-43e4-a96e-f602e8995304 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:18.358858627 +0000 UTC m=+64.691533243 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls") pod "image-registry-68f6f5b69b-h9jwj" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304") : secret "image-registry-tls" not found Sep 08 13:17:02.382602 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:02.382575 2554 generic.go:356] "Generic (PLEG): container finished" podID="311ed079-bea5-4b44-b238-fca763ef6788" containerID="0bb7b6d13dd03dd39fd10f7d509875a1fd8409357beaeb0ffcca090e30f16a9c" exitCode=0 Sep 08 13:17:02.382738 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:02.382644 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" event={"ID":"311ed079-bea5-4b44-b238-fca763ef6788","Type":"ContainerDied","Data":"0bb7b6d13dd03dd39fd10f7d509875a1fd8409357beaeb0ffcca090e30f16a9c"} Sep 08 13:17:02.384098 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:02.384074 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" event={"ID":"1f28adea-856b-4569-a468-44b8a0be3505","Type":"ContainerStarted","Data":"7f07fa4028dfd4264ef69bbbc43c0a2e99285433639d50bf24c31fbddde347c6"} Sep 08 13:17:02.384260 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:02.384247 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:17:02.385791 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:02.385759 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:17:02.425910 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:02.425863 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" podStartSLOduration=1.3122171790000001 podStartE2EDuration="11.425849537s" totalImagesPullingTime="10.113632358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:02.424787699 +0000 UTC m=+48.757462336" watchObservedRunningTime="2026-09-08 13:17:02.425849537 +0000 UTC m=+48.758524185" Sep 08 13:17:04.391266 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:04.391233 2554 generic.go:356] "Generic (PLEG): container finished" podID="311ed079-bea5-4b44-b238-fca763ef6788" containerID="9b02149b533fd69d23092a01dc813fbd1b30ef300773c6b4b6084299b8bd1c20" exitCode=0 Sep 08 13:17:04.391732 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:04.391303 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" event={"ID":"311ed079-bea5-4b44-b238-fca763ef6788","Type":"ContainerDied","Data":"9b02149b533fd69d23092a01dc813fbd1b30ef300773c6b4b6084299b8bd1c20"} Sep 08 13:17:09.407356 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:09.407319 2554 generic.go:356] "Generic (PLEG): container finished" podID="311ed079-bea5-4b44-b238-fca763ef6788" containerID="8e4b2bdb546f2288c1ca9c71f89b4ccc8f359fa96832deba8adbdc4cf7976c13" exitCode=0 Sep 08 13:17:09.407356 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:09.407357 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" event={"ID":"311ed079-bea5-4b44-b238-fca763ef6788","Type":"ContainerDied","Data":"8e4b2bdb546f2288c1ca9c71f89b4ccc8f359fa96832deba8adbdc4cf7976c13"} Sep 08 13:17:10.412372 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:10.412327 2554 generic.go:356] "Generic (PLEG): container finished" podID="311ed079-bea5-4b44-b238-fca763ef6788" containerID="19b3e640c9272c5a01ca32397dcae10396b235efb984128061862821e321c4f9" exitCode=0 Sep 08 13:17:10.412372 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:10.412371 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" event={"ID":"311ed079-bea5-4b44-b238-fca763ef6788","Type":"ContainerDied","Data":"19b3e640c9272c5a01ca32397dcae10396b235efb984128061862821e321c4f9"} Sep 08 13:17:11.419637 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:11.419603 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" event={"ID":"311ed079-bea5-4b44-b238-fca763ef6788","Type":"ContainerStarted","Data":"2df107cf4b1b92dbc3522c57d463187493741f9ceb5bdbd8f1c7392dbdeb4913"} Sep 08 13:17:11.442424 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:11.442374 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v6zgx" podStartSLOduration=6.571625395 podStartE2EDuration="57.442359505s" totalImagesPullingTime="50.87073411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-08 13:17:11.442293984 +0000 UTC m=+57.774968628" watchObservedRunningTime="2026-09-08 13:17:11.442359505 +0000 UTC m=+57.775034145" Sep 08 13:17:12.325321 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:12.325292 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxmjp" Sep 08 13:17:18.376782 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:18.376722 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:17:18.377185 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:18.376806 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:17:18.377185 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:18.376889 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 13:17:18.377185 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:18.376918 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 13:17:18.377185 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:18.376929 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f6f5b69b-h9jwj: secret "image-registry-tls" not found Sep 08 13:17:18.377185 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:18.376967 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls podName:19ee7d05-e185-443a-ba8e-deb5446dd2ab nodeName:}" failed. No retries permitted until 2026-09-08 13:17:50.376948813 +0000 UTC m=+96.709623432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls") pod "dns-default-w9xj9" (UID: "19ee7d05-e185-443a-ba8e-deb5446dd2ab") : secret "dns-default-metrics-tls" not found Sep 08 13:17:18.377185 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:18.376983 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls podName:41ab8e0e-1c37-43e4-a96e-f602e8995304 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:50.376976604 +0000 UTC m=+96.709651220 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls") pod "image-registry-68f6f5b69b-h9jwj" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304") : secret "image-registry-tls" not found Sep 08 13:17:19.988439 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:19.988404 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:17:19.988977 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:19.988466 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:17:19.991174 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:19.991158 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 13:17:19.991330 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:19.991309 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 13:17:19.998763 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:19.998740 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 13:17:19.998865 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:19.998823 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs podName:dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:23.998800361 +0000 UTC m=+130.331474982 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs") pod "network-metrics-daemon-sghnw" (UID: "dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3") : secret "metrics-daemon-secret" not found Sep 08 13:17:20.001088 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.001070 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02ad25a4-76db-4b91-b212-a701afcd24b8-original-pull-secret\") pod \"global-pull-secret-syncer-vl8zn\" (UID: \"02ad25a4-76db-4b91-b212-a701afcd24b8\") " pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:17:20.006574 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.006555 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl8zn" Sep 08 13:17:20.089617 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.089580 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:17:20.092451 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.092430 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 13:17:20.103736 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.103711 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 13:17:20.113441 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.113417 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdjp4\" (UniqueName: \"kubernetes.io/projected/c0eab916-82d1-4c16-8339-12cabd7e0d6c-kube-api-access-hdjp4\") pod \"network-check-target-q8rk6\" (UID: \"c0eab916-82d1-4c16-8339-12cabd7e0d6c\") " pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:17:20.280716 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.280628 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vl8zn"] Sep 08 13:17:20.285530 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:17:20.285467 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ad25a4_76db_4b91_b212_a701afcd24b8.slice/crio-27c8932b5709698d18f8fcd3594f3c6f153b962aa2ae6e61c84027de8eec60fa WatchSource:0}: Error finding container 27c8932b5709698d18f8fcd3594f3c6f153b962aa2ae6e61c84027de8eec60fa: Status 404 returned error can't find the container with id 27c8932b5709698d18f8fcd3594f3c6f153b962aa2ae6e61c84027de8eec60fa Sep 08 13:17:20.304454 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.304428 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7tdz6\"" Sep 08 13:17:20.313080 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.313059 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:17:20.446615 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.446581 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vl8zn" event={"ID":"02ad25a4-76db-4b91-b212-a701afcd24b8","Type":"ContainerStarted","Data":"27c8932b5709698d18f8fcd3594f3c6f153b962aa2ae6e61c84027de8eec60fa"} Sep 08 13:17:20.583142 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:20.583058 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8rk6"] Sep 08 13:17:20.586548 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:17:20.586520 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0eab916_82d1_4c16_8339_12cabd7e0d6c.slice/crio-7977abc49f2e3b0dbaf54f94eb68afe6e68b150c7e02ca1ac59cff0b96745513 WatchSource:0}: Error finding container 7977abc49f2e3b0dbaf54f94eb68afe6e68b150c7e02ca1ac59cff0b96745513: Status 404 returned error can't find the container with id 7977abc49f2e3b0dbaf54f94eb68afe6e68b150c7e02ca1ac59cff0b96745513 Sep 08 13:17:21.451609 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:21.451570 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8rk6" event={"ID":"c0eab916-82d1-4c16-8339-12cabd7e0d6c","Type":"ContainerStarted","Data":"7977abc49f2e3b0dbaf54f94eb68afe6e68b150c7e02ca1ac59cff0b96745513"} Sep 08 13:17:25.466179 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:25.466150 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8rk6" event={"ID":"c0eab916-82d1-4c16-8339-12cabd7e0d6c","Type":"ContainerStarted","Data":"33ac879443dc182b82af67e7d7e19fdadf1cdb345d8fd8f0572f0a84860d7e79"} Sep 08 13:17:25.466480 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:25.466277 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:17:25.487429 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:25.487384 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q8rk6" podStartSLOduration=66.733369743 podStartE2EDuration="1m11.487370791s" totalImagesPullingTime="4.754001048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:25.486772308 +0000 UTC m=+71.819446949" watchObservedRunningTime="2026-09-08 13:17:25.487370791 +0000 UTC m=+71.820045429" Sep 08 13:17:26.470529 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:26.470486 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vl8zn" event={"ID":"02ad25a4-76db-4b91-b212-a701afcd24b8","Type":"ContainerStarted","Data":"b0c55a9b446f372189032f43ebc3142935254f38a086c0a36b2bc659b37abb8d"} Sep 08 13:17:26.487238 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:26.487189 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vl8zn" podStartSLOduration=67.426426284 podStartE2EDuration="1m12.487173042s" totalImagesPullingTime="5.060746758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:26.486522315 +0000 UTC m=+72.819196966" watchObservedRunningTime="2026-09-08 13:17:26.487173042 +0000 UTC m=+72.819847720" Sep 08 13:17:50.417380 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:50.417295 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:17:50.417380 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:50.417345 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:17:50.417865 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:50.417455 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 13:17:50.417865 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:50.417465 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f6f5b69b-h9jwj: secret "image-registry-tls" not found Sep 08 13:17:50.417865 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:50.417515 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls podName:41ab8e0e-1c37-43e4-a96e-f602e8995304 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.41750234 +0000 UTC m=+160.750176956 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls") pod "image-registry-68f6f5b69b-h9jwj" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304") : secret "image-registry-tls" not found Sep 08 13:17:50.417865 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:50.417454 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 13:17:50.417865 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:17:50.417593 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls podName:19ee7d05-e185-443a-ba8e-deb5446dd2ab nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.417578642 +0000 UTC m=+160.750253257 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls") pod "dns-default-w9xj9" (UID: "19ee7d05-e185-443a-ba8e-deb5446dd2ab") : secret "dns-default-metrics-tls" not found Sep 08 13:17:56.472793 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:17:56.472762 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q8rk6" Sep 08 13:18:24.061136 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:18:24.061093 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:18:24.061631 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:24.061212 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 13:18:24.061631 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:24.061277 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs podName:dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:26.061261688 +0000 UTC m=+252.393936304 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs") pod "network-metrics-daemon-sghnw" (UID: "dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3") : secret "metrics-daemon-secret" not found Sep 08 13:18:49.474271 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:49.474211 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" podUID="41ab8e0e-1c37-43e4-a96e-f602e8995304" Sep 08 13:18:49.493391 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:49.493351 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-w9xj9" podUID="19ee7d05-e185-443a-ba8e-deb5446dd2ab" Sep 08 13:18:49.715781 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:18:49.715751 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:18:49.715978 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:18:49.715751 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9xj9" Sep 08 13:18:51.196337 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:51.196293 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sghnw" podUID="dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3" Sep 08 13:18:54.477810 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:18:54.477777 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:18:54.478223 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:18:54.477830 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") pod \"image-registry-68f6f5b69b-h9jwj\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:18:54.478223 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:54.477921 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 13:18:54.478223 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:54.477937 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f6f5b69b-h9jwj: secret "image-registry-tls" not found Sep 08 13:18:54.478223 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:54.477980 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls podName:41ab8e0e-1c37-43e4-a96e-f602e8995304 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:56.477966183 +0000 UTC m=+282.810640799 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls") pod "image-registry-68f6f5b69b-h9jwj" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304") : secret "image-registry-tls" not found Sep 08 13:18:54.478223 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:54.477919 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 13:18:54.478223 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:18:54.478049 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls podName:19ee7d05-e185-443a-ba8e-deb5446dd2ab nodeName:}" failed. No retries permitted until 2026-09-08 13:20:56.478033165 +0000 UTC m=+282.810707786 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls") pod "dns-default-w9xj9" (UID: "19ee7d05-e185-443a-ba8e-deb5446dd2ab") : secret "dns-default-metrics-tls" not found Sep 08 13:18:58.748275 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:18:58.748240 2554 generic.go:356] "Generic (PLEG): container finished" podID="7c0cb302-4ba3-417d-a7f0-c8d2c3766282" containerID="fa126ed629166eb52b4259d34b220347903b08e2681105bfdd28d66d64e891d2" exitCode=255 Sep 08 13:18:58.748731 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:18:58.748299 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" event={"ID":"7c0cb302-4ba3-417d-a7f0-c8d2c3766282","Type":"ContainerDied","Data":"fa126ed629166eb52b4259d34b220347903b08e2681105bfdd28d66d64e891d2"} Sep 08 13:18:58.748731 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:18:58.748567 2554 scope.go:117] "RemoveContainer" containerID="fa126ed629166eb52b4259d34b220347903b08e2681105bfdd28d66d64e891d2" Sep 08 13:18:59.752865 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:18:59.752831 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f55cc7c88-gm6qv" event={"ID":"7c0cb302-4ba3-417d-a7f0-c8d2c3766282","Type":"ContainerStarted","Data":"85710d7401fa329cbaf5285f1d85a3f926e2593b5a794797edafd2493cc858b4"} Sep 08 13:19:02.183006 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:02.182968 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:19:02.384634 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:02.384581 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" podUID="1f28adea-856b-4569-a468-44b8a0be3505" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.11:8000/readyz\": dial tcp 10.132.0.11:8000: connect: connection refused" Sep 08 13:19:02.764134 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:02.764099 2554 generic.go:356] "Generic (PLEG): container finished" podID="1f28adea-856b-4569-a468-44b8a0be3505" containerID="7f07fa4028dfd4264ef69bbbc43c0a2e99285433639d50bf24c31fbddde347c6" exitCode=1 Sep 08 13:19:02.764290 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:02.764170 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" event={"ID":"1f28adea-856b-4569-a468-44b8a0be3505","Type":"ContainerDied","Data":"7f07fa4028dfd4264ef69bbbc43c0a2e99285433639d50bf24c31fbddde347c6"} Sep 08 13:19:02.764510 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:02.764496 2554 scope.go:117] "RemoveContainer" containerID="7f07fa4028dfd4264ef69bbbc43c0a2e99285433639d50bf24c31fbddde347c6" Sep 08 13:19:03.769314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:03.769277 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" event={"ID":"1f28adea-856b-4569-a468-44b8a0be3505","Type":"ContainerStarted","Data":"411d56b9a0af759043ffcc07666de0192b4ae3e49f826dc591f965aa597177b8"} Sep 08 13:19:03.769745 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:03.769566 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:19:03.770175 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:03.770155 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7966b9f4b6-kh752" Sep 08 13:19:06.186522 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.186490 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vvcsd"] Sep 08 13:19:06.189376 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.189358 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.192016 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.191993 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 13:19:06.192133 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.191998 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 13:19:06.192133 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.192000 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 13:19:06.193019 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.193006 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-brrfw\"" Sep 08 13:19:06.193078 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.193063 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 13:19:06.202054 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.202031 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vvcsd"] Sep 08 13:19:06.266382 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.266310 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8jd\" (UniqueName: \"kubernetes.io/projected/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-api-access-6m8jd\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.266382 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.266346 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.266382 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.266366 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.266597 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.266426 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2cca6b92-400b-4f9e-9c99-0603fd88c093-crio-socket\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.266597 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.266512 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cca6b92-400b-4f9e-9c99-0603fd88c093-data-volume\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.367612 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.367566 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cca6b92-400b-4f9e-9c99-0603fd88c093-data-volume\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.367820 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.367651 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8jd\" (UniqueName: \"kubernetes.io/projected/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-api-access-6m8jd\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.367820 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.367732 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.367820 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:06.367765 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 13:19:06.367820 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.367799 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.368027 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:06.367834 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:06.867814088 +0000 UTC m=+173.200488709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : configmap "kube-rbac-proxy" not found Sep 08 13:19:06.368027 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.367873 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2cca6b92-400b-4f9e-9c99-0603fd88c093-crio-socket\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.368027 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:06.367885 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 13:19:06.368027 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:06.367935 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:06.8679177 +0000 UTC m=+173.200592336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : secret "insights-runtime-extractor-tls" not found Sep 08 13:19:06.368027 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.367951 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cca6b92-400b-4f9e-9c99-0603fd88c093-data-volume\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.368027 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.367963 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2cca6b92-400b-4f9e-9c99-0603fd88c093-crio-socket\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.378220 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.378190 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8jd\" (UniqueName: \"kubernetes.io/projected/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-api-access-6m8jd\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.871868 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.871827 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.871868 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:06.871867 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:06.872108 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:06.871949 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 13:19:06.872108 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:06.872003 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 13:19:06.872108 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:06.872018 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:07.872001582 +0000 UTC m=+174.204676199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : configmap "kube-rbac-proxy" not found Sep 08 13:19:06.872108 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:06.872043 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:07.872031681 +0000 UTC m=+174.204706297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : secret "insights-runtime-extractor-tls" not found Sep 08 13:19:07.879281 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:07.879239 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:07.879281 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:07.879280 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:07.879733 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:07.879363 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 13:19:07.879733 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:07.879431 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 13:19:07.879733 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:07.879434 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.879414733 +0000 UTC m=+176.212089362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : configmap "kube-rbac-proxy" not found Sep 08 13:19:07.879733 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:07.879476 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.879465262 +0000 UTC m=+176.212139885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : secret "insights-runtime-extractor-tls" not found Sep 08 13:19:07.932819 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:07.932793 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hp94q_19f87d80-4c14-4f36-bd27-852618164729/dns-node-resolver/0.log" Sep 08 13:19:09.332999 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:09.332973 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rgj2t_dc121a5f-2eee-4ca2-a204-fcefee21d9ff/node-ca/0.log" Sep 08 13:19:09.897038 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:09.896999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:09.897038 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:09.897042 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:09.897245 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:09.897134 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 13:19:09.897245 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:09.897179 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 13:19:09.897245 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:09.897207 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:13.897189935 +0000 UTC m=+180.229864551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : configmap "kube-rbac-proxy" not found Sep 08 13:19:09.897245 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:09.897222 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:13.897215839 +0000 UTC m=+180.229890454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : secret "insights-runtime-extractor-tls" not found Sep 08 13:19:10.133092 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:10.133064 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z69vl_c357275f-3687-4075-bffd-b8acd4f279c4/serve-healthcheck-canary/0.log" Sep 08 13:19:13.930444 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:13.930390 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:13.930444 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:13.930448 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:13.930919 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:13.930513 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 13:19:13.930919 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:13.930564 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 13:19:13.930919 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:13.930590 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:21.930568993 +0000 UTC m=+188.263243614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : configmap "kube-rbac-proxy" not found Sep 08 13:19:13.930919 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:13.930606 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:21.930598561 +0000 UTC m=+188.263273178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : secret "insights-runtime-extractor-tls" not found Sep 08 13:19:21.995965 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:21.995928 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:21.995965 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:21.995970 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:21.996472 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:21.996065 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 13:19:21.996472 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:21.996139 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls podName:2cca6b92-400b-4f9e-9c99-0603fd88c093 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:37.99612281 +0000 UTC m=+204.328797430 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vvcsd" (UID: "2cca6b92-400b-4f9e-9c99-0603fd88c093") : secret "insights-runtime-extractor-tls" not found Sep 08 13:19:21.996472 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:21.996429 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2cca6b92-400b-4f9e-9c99-0603fd88c093-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:38.015012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:38.014963 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:38.017263 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:38.017243 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2cca6b92-400b-4f9e-9c99-0603fd88c093-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vvcsd\" (UID: \"2cca6b92-400b-4f9e-9c99-0603fd88c093\") " pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:38.300028 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:38.299940 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-brrfw\"" Sep 08 13:19:38.307671 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:38.307648 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vvcsd" Sep 08 13:19:38.580600 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:38.580523 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vvcsd"] Sep 08 13:19:38.583707 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:19:38.583663 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cca6b92_400b_4f9e_9c99_0603fd88c093.slice/crio-fa5bef6a22ebb8801760621a990cf3d54d80e6fc1f2c102c9a3327b036156de8 WatchSource:0}: Error finding container fa5bef6a22ebb8801760621a990cf3d54d80e6fc1f2c102c9a3327b036156de8: Status 404 returned error can't find the container with id fa5bef6a22ebb8801760621a990cf3d54d80e6fc1f2c102c9a3327b036156de8 Sep 08 13:19:38.869242 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:38.869161 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vvcsd" event={"ID":"2cca6b92-400b-4f9e-9c99-0603fd88c093","Type":"ContainerStarted","Data":"a97b18cea9fc1e545036ba54fddda44b7a2f31bf62faaf49efdb79571d846406"} Sep 08 13:19:38.869242 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:38.869222 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vvcsd" event={"ID":"2cca6b92-400b-4f9e-9c99-0603fd88c093","Type":"ContainerStarted","Data":"fa5bef6a22ebb8801760621a990cf3d54d80e6fc1f2c102c9a3327b036156de8"} Sep 08 13:19:39.874501 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:39.874468 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vvcsd" event={"ID":"2cca6b92-400b-4f9e-9c99-0603fd88c093","Type":"ContainerStarted","Data":"6bc2a1ef5dcca84398a0914736c19b798a4051ddb3a2eb0f394d3cc80011e022"} Sep 08 13:19:40.879414 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:40.879332 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vvcsd" event={"ID":"2cca6b92-400b-4f9e-9c99-0603fd88c093","Type":"ContainerStarted","Data":"23c6880f72f4ed0826385e5af0b52eaa1d413af833ebe85794032dfd11295e2a"} Sep 08 13:19:40.900314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:40.900264 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vvcsd" podStartSLOduration=33.084121401 podStartE2EDuration="34.900246478s" totalImagesPullingTime="1.816125077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:19:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:40.898977065 +0000 UTC m=+207.231651703" watchObservedRunningTime="2026-09-08 13:19:40.900246478 +0000 UTC m=+207.232921116" Sep 08 13:19:47.294494 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.294461 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k8dm2"] Sep 08 13:19:47.297793 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.297774 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.300795 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.300765 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 13:19:47.300940 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.300920 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 13:19:47.301014 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.301002 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 13:19:47.301074 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.301017 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 13:19:47.302043 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.302025 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 13:19:47.302142 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.302026 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jflq8\"" Sep 08 13:19:47.302142 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.302030 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 13:19:47.386582 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.386550 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-wtmp\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.386582 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.386583 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-textfile\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.386795 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.386602 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-tls\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.386795 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.386637 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.386795 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.386720 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrmlx\" (UniqueName: \"kubernetes.io/projected/32307348-cbf9-4ccf-a66e-2080f561beb4-kube-api-access-hrmlx\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.386795 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.386758 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32307348-cbf9-4ccf-a66e-2080f561beb4-metrics-client-ca\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.386795 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.386773 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.386948 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.386814 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32307348-cbf9-4ccf-a66e-2080f561beb4-root\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.386948 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.386870 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32307348-cbf9-4ccf-a66e-2080f561beb4-sys\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487315 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487248 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrmlx\" (UniqueName: \"kubernetes.io/projected/32307348-cbf9-4ccf-a66e-2080f561beb4-kube-api-access-hrmlx\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487514 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487349 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32307348-cbf9-4ccf-a66e-2080f561beb4-metrics-client-ca\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487514 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487378 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487514 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487405 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32307348-cbf9-4ccf-a66e-2080f561beb4-root\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487662 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487527 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32307348-cbf9-4ccf-a66e-2080f561beb4-sys\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487662 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487575 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-wtmp\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487662 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487593 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-textfile\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487662 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487595 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32307348-cbf9-4ccf-a66e-2080f561beb4-root\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487662 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487615 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-tls\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487921 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487705 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.487921 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487784 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-wtmp\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.488015 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.487944 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-textfile\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.488090 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.488048 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32307348-cbf9-4ccf-a66e-2080f561beb4-sys\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.488222 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.488150 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32307348-cbf9-4ccf-a66e-2080f561beb4-metrics-client-ca\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.488222 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.488172 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.489924 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.489901 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.490037 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.489947 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32307348-cbf9-4ccf-a66e-2080f561beb4-node-exporter-tls\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.498642 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.498606 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrmlx\" (UniqueName: \"kubernetes.io/projected/32307348-cbf9-4ccf-a66e-2080f561beb4-kube-api-access-hrmlx\") pod \"node-exporter-k8dm2\" (UID: \"32307348-cbf9-4ccf-a66e-2080f561beb4\") " pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.606938 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.606851 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8dm2" Sep 08 13:19:47.617538 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:19:47.617514 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32307348_cbf9_4ccf_a66e_2080f561beb4.slice/crio-3e1769b4e4ed509411d6187da59b882bd72390180b92e1a5130e107b7f54b4ae WatchSource:0}: Error finding container 3e1769b4e4ed509411d6187da59b882bd72390180b92e1a5130e107b7f54b4ae: Status 404 returned error can't find the container with id 3e1769b4e4ed509411d6187da59b882bd72390180b92e1a5130e107b7f54b4ae Sep 08 13:19:47.901699 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:47.901588 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8dm2" event={"ID":"32307348-cbf9-4ccf-a66e-2080f561beb4","Type":"ContainerStarted","Data":"3e1769b4e4ed509411d6187da59b882bd72390180b92e1a5130e107b7f54b4ae"} Sep 08 13:19:48.906121 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:48.906035 2554 generic.go:356] "Generic (PLEG): container finished" podID="32307348-cbf9-4ccf-a66e-2080f561beb4" containerID="afc667f914d6048153fc556f341edb09664cd1589fd6364736e26a85031db96c" exitCode=0 Sep 08 13:19:48.906121 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:48.906077 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8dm2" event={"ID":"32307348-cbf9-4ccf-a66e-2080f561beb4","Type":"ContainerDied","Data":"afc667f914d6048153fc556f341edb09664cd1589fd6364736e26a85031db96c"} Sep 08 13:19:49.910967 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:49.910930 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8dm2" event={"ID":"32307348-cbf9-4ccf-a66e-2080f561beb4","Type":"ContainerStarted","Data":"dbf71ff4dcedc550ac039426e049e544ba9029557b51d83b4ca47d5f87b02d1c"} Sep 08 13:19:49.911442 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:49.911028 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8dm2" event={"ID":"32307348-cbf9-4ccf-a66e-2080f561beb4","Type":"ContainerStarted","Data":"7fd7bec5cf303629f42d9c0440ab53676f479ce03d2f176fb231c0300b8e225e"} Sep 08 13:19:49.962887 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:49.962838 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k8dm2" podStartSLOduration=1.9549283069999999 podStartE2EDuration="2.962824041s" totalImagesPullingTime="937.650251ms" totalInitContainerRuntime="70.245483ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:49.959067405 +0000 UTC m=+216.291742049" watchObservedRunningTime="2026-09-08 13:19:49.962824041 +0000 UTC m=+216.295498680" Sep 08 13:19:51.408525 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:51.408483 2554 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" podUID="427a1265-e971-45c0-ab4b-3d14ec110ac8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 08 13:19:56.117516 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:56.117483 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f6f5b69b-h9jwj"] Sep 08 13:19:56.117892 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:19:56.117752 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" podUID="41ab8e0e-1c37-43e4-a96e-f602e8995304" Sep 08 13:19:56.935817 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:56.935789 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:19:56.942426 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:56.942404 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:19:57.063483 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.063447 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-certificates\") pod \"41ab8e0e-1c37-43e4-a96e-f602e8995304\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " Sep 08 13:19:57.063483 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.063490 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-trusted-ca\") pod \"41ab8e0e-1c37-43e4-a96e-f602e8995304\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " Sep 08 13:19:57.063769 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.063514 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-installation-pull-secrets\") pod \"41ab8e0e-1c37-43e4-a96e-f602e8995304\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " Sep 08 13:19:57.063769 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.063532 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvjrs\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-kube-api-access-nvjrs\") pod \"41ab8e0e-1c37-43e4-a96e-f602e8995304\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " Sep 08 13:19:57.063769 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.063581 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-bound-sa-token\") pod \"41ab8e0e-1c37-43e4-a96e-f602e8995304\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " Sep 08 13:19:57.063769 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.063608 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41ab8e0e-1c37-43e4-a96e-f602e8995304-ca-trust-extracted\") pod \"41ab8e0e-1c37-43e4-a96e-f602e8995304\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " Sep 08 13:19:57.063769 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.063631 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-image-registry-private-configuration\") pod \"41ab8e0e-1c37-43e4-a96e-f602e8995304\" (UID: \"41ab8e0e-1c37-43e4-a96e-f602e8995304\") " Sep 08 13:19:57.064090 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.064058 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ab8e0e-1c37-43e4-a96e-f602e8995304-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "41ab8e0e-1c37-43e4-a96e-f602e8995304" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 13:19:57.064186 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.064098 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "41ab8e0e-1c37-43e4-a96e-f602e8995304" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:19:57.064186 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.064113 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "41ab8e0e-1c37-43e4-a96e-f602e8995304" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:19:57.066046 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.066012 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-kube-api-access-nvjrs" (OuterVolumeSpecName: "kube-api-access-nvjrs") pod "41ab8e0e-1c37-43e4-a96e-f602e8995304" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304"). InnerVolumeSpecName "kube-api-access-nvjrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 13:19:57.066165 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.066082 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "41ab8e0e-1c37-43e4-a96e-f602e8995304" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 13:19:57.066265 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.066244 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "41ab8e0e-1c37-43e4-a96e-f602e8995304" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:19:57.066301 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.066287 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "41ab8e0e-1c37-43e4-a96e-f602e8995304" (UID: "41ab8e0e-1c37-43e4-a96e-f602e8995304"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:19:57.164585 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.164530 2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-bound-sa-token\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:19:57.164585 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.164577 2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41ab8e0e-1c37-43e4-a96e-f602e8995304-ca-trust-extracted\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:19:57.164585 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.164588 2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-image-registry-private-configuration\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:19:57.164585 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.164597 2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-certificates\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:19:57.164585 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.164607 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41ab8e0e-1c37-43e4-a96e-f602e8995304-trusted-ca\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:19:57.165105 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.164615 2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41ab8e0e-1c37-43e4-a96e-f602e8995304-installation-pull-secrets\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:19:57.165105 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.164623 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvjrs\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-kube-api-access-nvjrs\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:19:57.938893 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.938863 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f6f5b69b-h9jwj" Sep 08 13:19:57.973186 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.973159 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f6f5b69b-h9jwj"] Sep 08 13:19:57.978830 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:57.978806 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68f6f5b69b-h9jwj"] Sep 08 13:19:58.072274 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:58.072246 2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41ab8e0e-1c37-43e4-a96e-f602e8995304-registry-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:19:58.187820 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:19:58.187784 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ab8e0e-1c37-43e4-a96e-f602e8995304" path="/var/lib/kubelet/pods/41ab8e0e-1c37-43e4-a96e-f602e8995304/volumes" Sep 08 13:20:01.408031 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:01.407988 2554 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" podUID="427a1265-e971-45c0-ab4b-3d14ec110ac8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 08 13:20:11.407977 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:11.407933 2554 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" podUID="427a1265-e971-45c0-ab4b-3d14ec110ac8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 08 13:20:11.408447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:11.408009 2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" Sep 08 13:20:11.408489 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:11.408441 2554 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"aaecc6987ddc90f990c0869860f9397c18fba9020d03a19491ecc1ff902390ea"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 08 13:20:11.408527 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:11.408489 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" podUID="427a1265-e971-45c0-ab4b-3d14ec110ac8" containerName="service-proxy" containerID="cri-o://aaecc6987ddc90f990c0869860f9397c18fba9020d03a19491ecc1ff902390ea" gracePeriod=30 Sep 08 13:20:11.982971 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:11.982932 2554 generic.go:356] "Generic (PLEG): container finished" podID="427a1265-e971-45c0-ab4b-3d14ec110ac8" containerID="aaecc6987ddc90f990c0869860f9397c18fba9020d03a19491ecc1ff902390ea" exitCode=2 Sep 08 13:20:11.983136 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:11.983003 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" event={"ID":"427a1265-e971-45c0-ab4b-3d14ec110ac8","Type":"ContainerDied","Data":"aaecc6987ddc90f990c0869860f9397c18fba9020d03a19491ecc1ff902390ea"} Sep 08 13:20:11.983136 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:11.983048 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbf7b898c-m5xpf" event={"ID":"427a1265-e971-45c0-ab4b-3d14ec110ac8","Type":"ContainerStarted","Data":"78509990ae9de2740f3c120008be2fa9d1c3221eb57cf6729bbcf9918edc027b"} Sep 08 13:20:26.079105 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:26.079056 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:20:26.081429 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:26.081407 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3-metrics-certs\") pod \"network-metrics-daemon-sghnw\" (UID: \"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3\") " pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:20:26.187199 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:26.187169 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lvcfq\"" Sep 08 13:20:26.194730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:26.194713 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sghnw" Sep 08 13:20:26.467347 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:26.467166 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sghnw"] Sep 08 13:20:26.469821 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:20:26.469794 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf3cfc3_2d6e_4243_aeed_a87ae931f4f3.slice/crio-da2917505c6be604aa3bc44f5d9c04c021a3190bcf672929fe3f2de6d70cb926 WatchSource:0}: Error finding container da2917505c6be604aa3bc44f5d9c04c021a3190bcf672929fe3f2de6d70cb926: Status 404 returned error can't find the container with id da2917505c6be604aa3bc44f5d9c04c021a3190bcf672929fe3f2de6d70cb926 Sep 08 13:20:27.033619 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:27.033576 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sghnw" event={"ID":"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3","Type":"ContainerStarted","Data":"da2917505c6be604aa3bc44f5d9c04c021a3190bcf672929fe3f2de6d70cb926"} Sep 08 13:20:28.038948 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:28.038912 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sghnw" event={"ID":"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3","Type":"ContainerStarted","Data":"dd909dee3ef926ef2800713a93ee5928e9a1756814fb5aa41b60a5ae94432b5f"} Sep 08 13:20:28.039384 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:28.038973 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sghnw" event={"ID":"dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3","Type":"ContainerStarted","Data":"8890169c1759b679ae0deda1e2587157f4ac45736788586d13b9e33f62adece1"} Sep 08 13:20:28.059776 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:28.059723 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sghnw" podStartSLOduration=253.289620459 podStartE2EDuration="4m14.059705269s" totalImagesPullingTime="770.08481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:28.057544374 +0000 UTC m=+254.390219025" watchObservedRunningTime="2026-09-08 13:20:28.059705269 +0000 UTC m=+254.392379915" Sep 08 13:20:52.716522 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:20:52.716478 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-w9xj9" podUID="19ee7d05-e185-443a-ba8e-deb5446dd2ab" Sep 08 13:20:53.119048 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:53.119019 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9xj9" Sep 08 13:20:56.497056 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:56.497014 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:20:56.499263 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:56.499241 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ee7d05-e185-443a-ba8e-deb5446dd2ab-metrics-tls\") pod \"dns-default-w9xj9\" (UID: \"19ee7d05-e185-443a-ba8e-deb5446dd2ab\") " pod="openshift-dns/dns-default-w9xj9" Sep 08 13:20:56.722910 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:56.722879 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6zj7\"" Sep 08 13:20:56.730974 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:56.730956 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9xj9" Sep 08 13:20:57.001129 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:57.001094 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w9xj9"] Sep 08 13:20:57.004993 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:20:57.004964 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ee7d05_e185_443a_ba8e_deb5446dd2ab.slice/crio-278125887fb2618d9d7c73513cdf76831e3d27329664e69331c370a2028a7378 WatchSource:0}: Error finding container 278125887fb2618d9d7c73513cdf76831e3d27329664e69331c370a2028a7378: Status 404 returned error can't find the container with id 278125887fb2618d9d7c73513cdf76831e3d27329664e69331c370a2028a7378 Sep 08 13:20:57.132193 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:20:57.132110 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9xj9" event={"ID":"19ee7d05-e185-443a-ba8e-deb5446dd2ab","Type":"ContainerStarted","Data":"278125887fb2618d9d7c73513cdf76831e3d27329664e69331c370a2028a7378"} Sep 08 13:21:02.151594 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:21:02.151554 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9xj9" event={"ID":"19ee7d05-e185-443a-ba8e-deb5446dd2ab","Type":"ContainerStarted","Data":"e0d219a6f9a9599a947642a28afb0aee017ca326b5cd42bd60b8e0333ca4a91d"} Sep 08 13:21:02.152041 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:21:02.151613 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9xj9" event={"ID":"19ee7d05-e185-443a-ba8e-deb5446dd2ab","Type":"ContainerStarted","Data":"c2b3015c8b44f1cddb6a4585347ee4fef380437ab7561c5beb36879551d7dbad"} Sep 08 13:21:02.152041 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:21:02.151665 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w9xj9" Sep 08 13:21:02.174830 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:21:02.174782 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w9xj9" podStartSLOduration=251.683993772 podStartE2EDuration="4m16.174766586s" totalImagesPullingTime="4.490772814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:02.173025418 +0000 UTC m=+288.505700057" watchObservedRunningTime="2026-09-08 13:21:02.174766586 +0000 UTC m=+288.507441225" Sep 08 13:21:12.158252 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:21:12.158219 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w9xj9" Sep 08 13:21:14.096238 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:21:14.096195 2554 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 13:24:25.514415 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.514378 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7"] Sep 08 13:24:25.517315 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.517297 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:25.521911 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.521875 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 08 13:24:25.521911 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.521898 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-l8bh4\"" Sep 08 13:24:25.522095 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.521905 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 08 13:24:25.522095 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.521943 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 08 13:24:25.522095 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.521992 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 08 13:24:25.522095 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.521887 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Sep 08 13:24:25.528273 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.528253 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7"] Sep 08 13:24:25.618984 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.618953 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e81a00a5-8799-4bdd-8092-b49bd30aedcf-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:25.619131 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.619006 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2tl\" (UniqueName: \"kubernetes.io/projected/e81a00a5-8799-4bdd-8092-b49bd30aedcf-kube-api-access-4x2tl\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:25.619131 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.619095 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e81a00a5-8799-4bdd-8092-b49bd30aedcf-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:25.720402 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.720370 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2tl\" (UniqueName: \"kubernetes.io/projected/e81a00a5-8799-4bdd-8092-b49bd30aedcf-kube-api-access-4x2tl\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:25.720571 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.720415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e81a00a5-8799-4bdd-8092-b49bd30aedcf-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:25.720571 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.720436 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e81a00a5-8799-4bdd-8092-b49bd30aedcf-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:25.720571 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:24:25.720552 2554 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found Sep 08 13:24:25.720746 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:24:25.720582 2554 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7: secret "keda-metrics-apiserver-certs" not found Sep 08 13:24:25.720746 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:24:25.720650 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e81a00a5-8799-4bdd-8092-b49bd30aedcf-certificates podName:e81a00a5-8799-4bdd-8092-b49bd30aedcf nodeName:}" failed. No retries permitted until 2026-09-08 13:24:26.220626669 +0000 UTC m=+492.553301304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e81a00a5-8799-4bdd-8092-b49bd30aedcf-certificates") pod "keda-metrics-apiserver-77857b7b98-wcrs7" (UID: "e81a00a5-8799-4bdd-8092-b49bd30aedcf") : secret "keda-metrics-apiserver-certs" not found Sep 08 13:24:25.720884 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.720846 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e81a00a5-8799-4bdd-8092-b49bd30aedcf-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:25.730209 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:25.730182 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2tl\" (UniqueName: \"kubernetes.io/projected/e81a00a5-8799-4bdd-8092-b49bd30aedcf-kube-api-access-4x2tl\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:26.224696 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:26.224658 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e81a00a5-8799-4bdd-8092-b49bd30aedcf-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:26.227113 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:26.227088 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e81a00a5-8799-4bdd-8092-b49bd30aedcf-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wcrs7\" (UID: \"e81a00a5-8799-4bdd-8092-b49bd30aedcf\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:26.426909 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:26.426873 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:26.703125 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:26.703093 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7"] Sep 08 13:24:26.706461 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:24:26.706430 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81a00a5_8799_4bdd_8092_b49bd30aedcf.slice/crio-7c5265926486a7cefc167a4e3df915d645c40d249bce2bd393175196f6163694 WatchSource:0}: Error finding container 7c5265926486a7cefc167a4e3df915d645c40d249bce2bd393175196f6163694: Status 404 returned error can't find the container with id 7c5265926486a7cefc167a4e3df915d645c40d249bce2bd393175196f6163694 Sep 08 13:24:26.707791 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:26.707775 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 13:24:26.804098 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:26.804066 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" event={"ID":"e81a00a5-8799-4bdd-8092-b49bd30aedcf","Type":"ContainerStarted","Data":"7c5265926486a7cefc167a4e3df915d645c40d249bce2bd393175196f6163694"} Sep 08 13:24:29.816357 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:29.816315 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" event={"ID":"e81a00a5-8799-4bdd-8092-b49bd30aedcf","Type":"ContainerStarted","Data":"232001347e19069f93af3cb6974d1ee35ebf276141fccc4fe92f3c50becda4b0"} Sep 08 13:24:29.816842 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:29.816430 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:40.824066 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:40.824037 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" Sep 08 13:24:40.845221 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:24:40.845168 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wcrs7" podStartSLOduration=13.097042083 podStartE2EDuration="15.845152108s" totalImagesPullingTime="2.748110025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:29.841253859 +0000 UTC m=+496.173928497" watchObservedRunningTime="2026-09-08 13:24:40.845152108 +0000 UTC m=+507.177826751" Sep 08 13:25:45.840211 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.840177 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79"] Sep 08 13:25:45.843132 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.843115 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:25:45.846738 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.846713 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 08 13:25:45.847867 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.847852 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 08 13:25:45.847949 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.847887 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Sep 08 13:25:45.847949 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.847909 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-qb2sl\"" Sep 08 13:25:45.853501 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.853479 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79"] Sep 08 13:25:45.892913 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.892885 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34a620f1-8b55-4418-afb7-931392727606-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-6wc79\" (UID: \"34a620f1-8b55-4418-afb7-931392727606\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:25:45.893022 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.892931 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbkn\" (UniqueName: \"kubernetes.io/projected/34a620f1-8b55-4418-afb7-931392727606-kube-api-access-6nbkn\") pod \"llmisvc-controller-manager-6f9bbc7b7c-6wc79\" (UID: \"34a620f1-8b55-4418-afb7-931392727606\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:25:45.993230 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.993196 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34a620f1-8b55-4418-afb7-931392727606-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-6wc79\" (UID: \"34a620f1-8b55-4418-afb7-931392727606\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:25:45.993372 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.993248 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbkn\" (UniqueName: \"kubernetes.io/projected/34a620f1-8b55-4418-afb7-931392727606-kube-api-access-6nbkn\") pod \"llmisvc-controller-manager-6f9bbc7b7c-6wc79\" (UID: \"34a620f1-8b55-4418-afb7-931392727606\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:25:45.995601 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:45.995551 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34a620f1-8b55-4418-afb7-931392727606-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-6wc79\" (UID: \"34a620f1-8b55-4418-afb7-931392727606\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:25:46.000934 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:46.000908 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbkn\" (UniqueName: \"kubernetes.io/projected/34a620f1-8b55-4418-afb7-931392727606-kube-api-access-6nbkn\") pod \"llmisvc-controller-manager-6f9bbc7b7c-6wc79\" (UID: \"34a620f1-8b55-4418-afb7-931392727606\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:25:46.152660 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:46.152583 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:25:46.420350 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:46.420322 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79"] Sep 08 13:25:46.422392 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:25:46.422364 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod34a620f1_8b55_4418_afb7_931392727606.slice/crio-86bb1fde3da5f2d4d23e9badc15e94be8136ac9071652ed08cf9c5ed1700ac63 WatchSource:0}: Error finding container 86bb1fde3da5f2d4d23e9badc15e94be8136ac9071652ed08cf9c5ed1700ac63: Status 404 returned error can't find the container with id 86bb1fde3da5f2d4d23e9badc15e94be8136ac9071652ed08cf9c5ed1700ac63 Sep 08 13:25:47.078449 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:47.078410 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" event={"ID":"34a620f1-8b55-4418-afb7-931392727606","Type":"ContainerStarted","Data":"86bb1fde3da5f2d4d23e9badc15e94be8136ac9071652ed08cf9c5ed1700ac63"} Sep 08 13:25:49.086345 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:49.086316 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" event={"ID":"34a620f1-8b55-4418-afb7-931392727606","Type":"ContainerStarted","Data":"353919a7fa55f46694f6b10b6c9790a76388eee13cf9224205b43729d398fc54"} Sep 08 13:25:49.086730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:49.086387 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:25:49.103634 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:25:49.103589 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" podStartSLOduration=1.9347994819999998 podStartE2EDuration="4.10357379s" totalImagesPullingTime="2.168774308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:25:49.10194501 +0000 UTC m=+575.434619647" watchObservedRunningTime="2026-09-08 13:25:49.10357379 +0000 UTC m=+575.436248446" Sep 08 13:26:20.092237 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:26:20.092203 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-6wc79" Sep 08 13:27:05.252521 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.252491 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-bxh55"] Sep 08 13:27:05.255748 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.255727 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:05.258502 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.258482 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 08 13:27:05.258613 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.258520 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-vzbch\"" Sep 08 13:27:05.264446 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.264425 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bxh55"] Sep 08 13:27:05.351402 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.351362 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28js\" (UniqueName: \"kubernetes.io/projected/be21d695-e76e-4e91-aecc-4c4d498e8662-kube-api-access-b28js\") pod \"model-serving-api-7b954ffc95-bxh55\" (UID: \"be21d695-e76e-4e91-aecc-4c4d498e8662\") " pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:05.351591 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.351406 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be21d695-e76e-4e91-aecc-4c4d498e8662-tls-certs\") pod \"model-serving-api-7b954ffc95-bxh55\" (UID: \"be21d695-e76e-4e91-aecc-4c4d498e8662\") " pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:05.452419 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.452383 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be21d695-e76e-4e91-aecc-4c4d498e8662-tls-certs\") pod \"model-serving-api-7b954ffc95-bxh55\" (UID: \"be21d695-e76e-4e91-aecc-4c4d498e8662\") " pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:05.452592 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.452468 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28js\" (UniqueName: \"kubernetes.io/projected/be21d695-e76e-4e91-aecc-4c4d498e8662-kube-api-access-b28js\") pod \"model-serving-api-7b954ffc95-bxh55\" (UID: \"be21d695-e76e-4e91-aecc-4c4d498e8662\") " pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:05.452592 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:27:05.452554 2554 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 08 13:27:05.452667 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:27:05.452624 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be21d695-e76e-4e91-aecc-4c4d498e8662-tls-certs podName:be21d695-e76e-4e91-aecc-4c4d498e8662 nodeName:}" failed. No retries permitted until 2026-09-08 13:27:05.952605415 +0000 UTC m=+652.285280051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be21d695-e76e-4e91-aecc-4c4d498e8662-tls-certs") pod "model-serving-api-7b954ffc95-bxh55" (UID: "be21d695-e76e-4e91-aecc-4c4d498e8662") : secret "model-serving-api-tls" not found Sep 08 13:27:05.460723 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.460699 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28js\" (UniqueName: \"kubernetes.io/projected/be21d695-e76e-4e91-aecc-4c4d498e8662-kube-api-access-b28js\") pod \"model-serving-api-7b954ffc95-bxh55\" (UID: \"be21d695-e76e-4e91-aecc-4c4d498e8662\") " pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:05.956360 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.956321 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be21d695-e76e-4e91-aecc-4c4d498e8662-tls-certs\") pod \"model-serving-api-7b954ffc95-bxh55\" (UID: \"be21d695-e76e-4e91-aecc-4c4d498e8662\") " pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:05.958720 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:05.958668 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be21d695-e76e-4e91-aecc-4c4d498e8662-tls-certs\") pod \"model-serving-api-7b954ffc95-bxh55\" (UID: \"be21d695-e76e-4e91-aecc-4c4d498e8662\") " pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:06.165794 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:06.165761 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:06.433896 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:06.433845 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bxh55"] Sep 08 13:27:06.436773 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:27:06.436746 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe21d695_e76e_4e91_aecc_4c4d498e8662.slice/crio-46c6ceb85938b942b3a20f3a298946eab062455b2c8fbfe9f46fd344e1033368 WatchSource:0}: Error finding container 46c6ceb85938b942b3a20f3a298946eab062455b2c8fbfe9f46fd344e1033368: Status 404 returned error can't find the container with id 46c6ceb85938b942b3a20f3a298946eab062455b2c8fbfe9f46fd344e1033368 Sep 08 13:27:07.341518 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:07.341483 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bxh55" event={"ID":"be21d695-e76e-4e91-aecc-4c4d498e8662","Type":"ContainerStarted","Data":"46c6ceb85938b942b3a20f3a298946eab062455b2c8fbfe9f46fd344e1033368"} Sep 08 13:27:10.353407 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:10.353326 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bxh55" event={"ID":"be21d695-e76e-4e91-aecc-4c4d498e8662","Type":"ContainerStarted","Data":"617b44bf260edcc0d293195e1835f87aa936afeb074e4be4641f201e99f87121"} Sep 08 13:27:10.353778 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:10.353437 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:27:10.371764 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:10.371714 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-bxh55" podStartSLOduration=1.732276815 podStartE2EDuration="5.37169869s" totalImagesPullingTime="3.639421875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:10.370678365 +0000 UTC m=+656.703353003" watchObservedRunningTime="2026-09-08 13:27:10.37169869 +0000 UTC m=+656.704373349" Sep 08 13:27:21.361773 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:27:21.361739 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-bxh55" Sep 08 13:28:58.478767 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.478726 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk"] Sep 08 13:28:58.481955 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.481939 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.485170 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.485150 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Sep 08 13:28:58.485170 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.485156 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Sep 08 13:28:58.485498 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.485484 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pw8wg\"" Sep 08 13:28:58.485574 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.485508 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 08 13:28:58.487498 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.486362 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 08 13:28:58.490065 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.489403 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 08 13:28:58.490065 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.489791 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 08 13:28:58.496638 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.496609 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk"] Sep 08 13:28:58.553625 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.553586 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.553811 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.553636 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d022bb-2968-40e9-932f-fe49a9c2885e-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.553811 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.553718 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.553811 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.553741 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d022bb-2968-40e9-932f-fe49a9c2885e-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.553811 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.553757 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wwzx\" (UniqueName: \"kubernetes.io/projected/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-api-access-4wwzx\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.654417 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.654372 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.654417 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.654419 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d022bb-2968-40e9-932f-fe49a9c2885e-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.654740 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.654439 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wwzx\" (UniqueName: \"kubernetes.io/projected/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-api-access-4wwzx\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.654740 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.654459 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.654740 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:28:58.654555 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Sep 08 13:28:58.654740 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.654583 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d022bb-2968-40e9-932f-fe49a9c2885e-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.654740 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:28:58.654639 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74d022bb-2968-40e9-932f-fe49a9c2885e-proxy-tls podName:74d022bb-2968-40e9-932f-fe49a9c2885e nodeName:}" failed. No retries permitted until 2026-09-08 13:28:59.154618817 +0000 UTC m=+765.487293448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/74d022bb-2968-40e9-932f-fe49a9c2885e-proxy-tls") pod "isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" (UID: "74d022bb-2968-40e9-932f-fe49a9c2885e") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Sep 08 13:28:58.654964 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.654943 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d022bb-2968-40e9-932f-fe49a9c2885e-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.655106 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.655088 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.655156 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.655142 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:58.665087 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:58.665060 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wwzx\" (UniqueName: \"kubernetes.io/projected/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-api-access-4wwzx\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:59.158275 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:59.158219 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d022bb-2968-40e9-932f-fe49a9c2885e-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:59.160762 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:59.160736 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d022bb-2968-40e9-932f-fe49a9c2885e-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-766ff5986c-vkntk\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:59.394747 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:59.394710 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:28:59.664331 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:59.664307 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk"] Sep 08 13:28:59.666376 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:28:59.666340 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d022bb_2968_40e9_932f_fe49a9c2885e.slice/crio-563597ea30fb9860badc9336822468230462d37d66967fcb8a003a2402196d2d WatchSource:0}: Error finding container 563597ea30fb9860badc9336822468230462d37d66967fcb8a003a2402196d2d: Status 404 returned error can't find the container with id 563597ea30fb9860badc9336822468230462d37d66967fcb8a003a2402196d2d Sep 08 13:28:59.720435 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:28:59.720360 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" event={"ID":"74d022bb-2968-40e9-932f-fe49a9c2885e","Type":"ContainerStarted","Data":"563597ea30fb9860badc9336822468230462d37d66967fcb8a003a2402196d2d"} Sep 08 13:29:03.736325 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:03.736287 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" event={"ID":"74d022bb-2968-40e9-932f-fe49a9c2885e","Type":"ContainerStarted","Data":"969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6"} Sep 08 13:29:06.748860 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:06.748820 2554 generic.go:356] "Generic (PLEG): container finished" podID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerID="969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6" exitCode=0 Sep 08 13:29:06.749231 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:06.748867 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" event={"ID":"74d022bb-2968-40e9-932f-fe49a9c2885e","Type":"ContainerDied","Data":"969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6"} Sep 08 13:29:19.808098 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:19.808061 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" event={"ID":"74d022bb-2968-40e9-932f-fe49a9c2885e","Type":"ContainerStarted","Data":"1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c"} Sep 08 13:29:22.826549 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:22.826508 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" event={"ID":"74d022bb-2968-40e9-932f-fe49a9c2885e","Type":"ContainerStarted","Data":"0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5"} Sep 08 13:29:22.826941 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:22.826620 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:29:22.848737 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:22.848672 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podStartSLOduration=2.538112854 podStartE2EDuration="24.84865716s" totalImagesPullingTime="18.437628877s" totalInitContainerRuntime="3.872915429s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:22.846791064 +0000 UTC m=+789.179465708" watchObservedRunningTime="2026-09-08 13:29:22.84865716 +0000 UTC m=+789.181331797" Sep 08 13:29:23.829739 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:23.829704 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:29:23.830752 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:23.830729 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:29:24.833129 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:24.833090 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:29:29.838093 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:29.838065 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:29:29.838713 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:29.838663 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:29:39.839646 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:39.839561 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:29:49.838584 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:49.838546 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:29:59.839360 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:29:59.839322 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:30:09.839202 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:09.839155 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:30:17.791733 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.791696 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4"] Sep 08 13:30:17.794627 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.794609 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:17.797199 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.797179 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f794f-kube-rbac-proxy-sar-config\"" Sep 08 13:30:17.797314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.797180 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f794f-serving-cert\"" Sep 08 13:30:17.806010 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.805984 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4"] Sep 08 13:30:17.858102 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.858071 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f84bd86-43cc-4f30-b38a-180837aba088-openshift-service-ca-bundle\") pod \"switch-graph-f794f-6766495c86-m57t4\" (UID: \"3f84bd86-43cc-4f30-b38a-180837aba088\") " pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:17.858253 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.858131 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f84bd86-43cc-4f30-b38a-180837aba088-proxy-tls\") pod \"switch-graph-f794f-6766495c86-m57t4\" (UID: \"3f84bd86-43cc-4f30-b38a-180837aba088\") " pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:17.958694 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.958652 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f84bd86-43cc-4f30-b38a-180837aba088-proxy-tls\") pod \"switch-graph-f794f-6766495c86-m57t4\" (UID: \"3f84bd86-43cc-4f30-b38a-180837aba088\") " pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:17.958864 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.958737 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f84bd86-43cc-4f30-b38a-180837aba088-openshift-service-ca-bundle\") pod \"switch-graph-f794f-6766495c86-m57t4\" (UID: \"3f84bd86-43cc-4f30-b38a-180837aba088\") " pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:17.959326 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.959308 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f84bd86-43cc-4f30-b38a-180837aba088-openshift-service-ca-bundle\") pod \"switch-graph-f794f-6766495c86-m57t4\" (UID: \"3f84bd86-43cc-4f30-b38a-180837aba088\") " pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:17.960999 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:17.960967 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f84bd86-43cc-4f30-b38a-180837aba088-proxy-tls\") pod \"switch-graph-f794f-6766495c86-m57t4\" (UID: \"3f84bd86-43cc-4f30-b38a-180837aba088\") " pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:18.104789 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:18.104702 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:18.381011 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:18.380925 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4"] Sep 08 13:30:18.383753 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:30:18.383719 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f84bd86_43cc_4f30_b38a_180837aba088.slice/crio-f3fb241f40d63a66b0d117946887f4ea56191d49bebb20e300380ab9c4fee758 WatchSource:0}: Error finding container f3fb241f40d63a66b0d117946887f4ea56191d49bebb20e300380ab9c4fee758: Status 404 returned error can't find the container with id f3fb241f40d63a66b0d117946887f4ea56191d49bebb20e300380ab9c4fee758 Sep 08 13:30:18.385677 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:18.385658 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 13:30:19.019897 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:19.019862 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" event={"ID":"3f84bd86-43cc-4f30-b38a-180837aba088","Type":"ContainerStarted","Data":"f3fb241f40d63a66b0d117946887f4ea56191d49bebb20e300380ab9c4fee758"} Sep 08 13:30:19.838579 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:19.838543 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:30:24.038789 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:24.038699 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" event={"ID":"3f84bd86-43cc-4f30-b38a-180837aba088","Type":"ContainerStarted","Data":"4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86"} Sep 08 13:30:24.039250 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:24.038788 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:24.060092 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:24.060039 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" podStartSLOduration=1.6630278779999998 podStartE2EDuration="7.060024769s" totalImagesPullingTime="5.396996891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:24.057151018 +0000 UTC m=+850.389825655" watchObservedRunningTime="2026-09-08 13:30:24.060024769 +0000 UTC m=+850.392699426" Sep 08 13:30:29.839622 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:29.839578 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:30:30.047638 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:30.047609 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:31.932652 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:31.932615 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4"] Sep 08 13:30:31.933056 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:31.932803 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" containerID="cri-o://4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86" gracePeriod=30 Sep 08 13:30:35.046300 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:35.046246 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:30:39.839854 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:39.839823 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:30:40.045936 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:40.045899 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:30:45.046127 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:45.046084 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:30:45.046519 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:45.046194 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:30:50.046342 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:50.046301 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:30:55.045559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:55.045514 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:30:57.745981 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.745945 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s"] Sep 08 13:30:57.752108 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.752091 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:30:57.754669 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.754647 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Sep 08 13:30:57.754810 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.754652 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Sep 08 13:30:57.757800 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.757778 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s"] Sep 08 13:30:57.759283 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.759263 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-proxy-tls\") pod \"model-chainer-546cb8556c-kwp6s\" (UID: \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\") " pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:30:57.759374 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.759301 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-openshift-service-ca-bundle\") pod \"model-chainer-546cb8556c-kwp6s\" (UID: \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\") " pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:30:57.860138 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.860110 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-openshift-service-ca-bundle\") pod \"model-chainer-546cb8556c-kwp6s\" (UID: \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\") " pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:30:57.860319 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.860217 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-proxy-tls\") pod \"model-chainer-546cb8556c-kwp6s\" (UID: \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\") " pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:30:57.860396 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:30:57.860343 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Sep 08 13:30:57.860449 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:30:57.860422 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-proxy-tls podName:c281bc12-19c2-4af6-874d-0c23e4b1d9a1 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:58.360400166 +0000 UTC m=+884.693074783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-proxy-tls") pod "model-chainer-546cb8556c-kwp6s" (UID: "c281bc12-19c2-4af6-874d-0c23e4b1d9a1") : secret "model-chainer-serving-cert" not found Sep 08 13:30:57.860748 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:57.860717 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-openshift-service-ca-bundle\") pod \"model-chainer-546cb8556c-kwp6s\" (UID: \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\") " pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:30:58.363773 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:58.363739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-proxy-tls\") pod \"model-chainer-546cb8556c-kwp6s\" (UID: \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\") " pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:30:58.366238 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:58.366209 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-proxy-tls\") pod \"model-chainer-546cb8556c-kwp6s\" (UID: \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\") " pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:30:58.662694 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:58.662597 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:30:58.933137 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:58.933063 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s"] Sep 08 13:30:58.936196 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:30:58.936171 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc281bc12_19c2_4af6_874d_0c23e4b1d9a1.slice/crio-5b677b92c96e46b6a919686debf453a9bc2318e29a829317a40e9c65987dbef5 WatchSource:0}: Error finding container 5b677b92c96e46b6a919686debf453a9bc2318e29a829317a40e9c65987dbef5: Status 404 returned error can't find the container with id 5b677b92c96e46b6a919686debf453a9bc2318e29a829317a40e9c65987dbef5 Sep 08 13:30:59.161356 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:59.161323 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" event={"ID":"c281bc12-19c2-4af6-874d-0c23e4b1d9a1","Type":"ContainerStarted","Data":"0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8"} Sep 08 13:30:59.161505 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:59.161378 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" event={"ID":"c281bc12-19c2-4af6-874d-0c23e4b1d9a1","Type":"ContainerStarted","Data":"5b677b92c96e46b6a919686debf453a9bc2318e29a829317a40e9c65987dbef5"} Sep 08 13:30:59.161505 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:30:59.161410 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:31:00.045530 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:00.045492 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:31:02.081279 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.081252 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:31:02.094219 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.094194 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f84bd86-43cc-4f30-b38a-180837aba088-proxy-tls\") pod \"3f84bd86-43cc-4f30-b38a-180837aba088\" (UID: \"3f84bd86-43cc-4f30-b38a-180837aba088\") " Sep 08 13:31:02.094323 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.094255 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f84bd86-43cc-4f30-b38a-180837aba088-openshift-service-ca-bundle\") pod \"3f84bd86-43cc-4f30-b38a-180837aba088\" (UID: \"3f84bd86-43cc-4f30-b38a-180837aba088\") " Sep 08 13:31:02.094608 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.094587 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f84bd86-43cc-4f30-b38a-180837aba088-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3f84bd86-43cc-4f30-b38a-180837aba088" (UID: "3f84bd86-43cc-4f30-b38a-180837aba088"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:31:02.096400 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.096380 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f84bd86-43cc-4f30-b38a-180837aba088-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f84bd86-43cc-4f30-b38a-180837aba088" (UID: "3f84bd86-43cc-4f30-b38a-180837aba088"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:31:02.101064 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.101019 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" podStartSLOduration=5.101004305 podStartE2EDuration="5.101004305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:59.178619485 +0000 UTC m=+885.511294123" watchObservedRunningTime="2026-09-08 13:31:02.101004305 +0000 UTC m=+888.433678942" Sep 08 13:31:02.173566 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.173525 2554 generic.go:356] "Generic (PLEG): container finished" podID="3f84bd86-43cc-4f30-b38a-180837aba088" containerID="4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86" exitCode=0 Sep 08 13:31:02.173741 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.173583 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" Sep 08 13:31:02.173741 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.173619 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" event={"ID":"3f84bd86-43cc-4f30-b38a-180837aba088","Type":"ContainerDied","Data":"4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86"} Sep 08 13:31:02.173741 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.173673 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4" event={"ID":"3f84bd86-43cc-4f30-b38a-180837aba088","Type":"ContainerDied","Data":"f3fb241f40d63a66b0d117946887f4ea56191d49bebb20e300380ab9c4fee758"} Sep 08 13:31:02.173741 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.173707 2554 scope.go:117] "RemoveContainer" containerID="4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86" Sep 08 13:31:02.181899 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.181880 2554 scope.go:117] "RemoveContainer" containerID="4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86" Sep 08 13:31:02.182155 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:31:02.182138 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86\": container with ID starting with 4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86 not found: ID does not exist" containerID="4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86" Sep 08 13:31:02.182210 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.182164 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86"} err="failed to get container status \"4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86\": rpc error: code = NotFound desc = could not find container \"4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86\": container with ID starting with 4792c7f5ac918cd1fb338cec95881e93489242afe23a9233bdd90b604cb68f86 not found: ID does not exist" Sep 08 13:31:02.194094 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.194040 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4"] Sep 08 13:31:02.195153 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.195134 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f84bd86-43cc-4f30-b38a-180837aba088-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:31:02.195153 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.195152 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f84bd86-43cc-4f30-b38a-180837aba088-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:31:02.198044 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:02.198023 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f794f-6766495c86-m57t4"] Sep 08 13:31:04.188584 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:04.188539 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" path="/var/lib/kubelet/pods/3f84bd86-43cc-4f30-b38a-180837aba088/volumes" Sep 08 13:31:05.171445 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:05.171419 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:31:07.847938 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:07.847855 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s"] Sep 08 13:31:07.848365 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:07.848061 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" containerID="cri-o://0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8" gracePeriod=30 Sep 08 13:31:07.954965 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:07.954921 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk"] Sep 08 13:31:07.955243 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:07.955205 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" containerID="cri-o://1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c" gracePeriod=30 Sep 08 13:31:07.955391 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:07.955265 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kube-rbac-proxy" containerID="cri-o://0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5" gracePeriod=30 Sep 08 13:31:08.197959 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:08.197879 2554 generic.go:356] "Generic (PLEG): container finished" podID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerID="0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5" exitCode=2 Sep 08 13:31:08.197959 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:08.197925 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" event={"ID":"74d022bb-2968-40e9-932f-fe49a9c2885e","Type":"ContainerDied","Data":"0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5"} Sep 08 13:31:09.833386 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:09.833343 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused" Sep 08 13:31:09.838675 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:09.838645 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused" Sep 08 13:31:10.171725 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:10.171610 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:31:12.786600 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.786576 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:31:12.875797 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.875704 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d022bb-2968-40e9-932f-fe49a9c2885e-proxy-tls\") pod \"74d022bb-2968-40e9-932f-fe49a9c2885e\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " Sep 08 13:31:12.875797 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.875743 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-rbac-proxy-sar-config\") pod \"74d022bb-2968-40e9-932f-fe49a9c2885e\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " Sep 08 13:31:12.875797 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.875766 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d022bb-2968-40e9-932f-fe49a9c2885e-kserve-provision-location\") pod \"74d022bb-2968-40e9-932f-fe49a9c2885e\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " Sep 08 13:31:12.876080 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.875896 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-cabundle-cert\") pod \"74d022bb-2968-40e9-932f-fe49a9c2885e\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " Sep 08 13:31:12.876080 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.875934 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wwzx\" (UniqueName: \"kubernetes.io/projected/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-api-access-4wwzx\") pod \"74d022bb-2968-40e9-932f-fe49a9c2885e\" (UID: \"74d022bb-2968-40e9-932f-fe49a9c2885e\") " Sep 08 13:31:12.876203 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.876179 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d022bb-2968-40e9-932f-fe49a9c2885e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74d022bb-2968-40e9-932f-fe49a9c2885e" (UID: "74d022bb-2968-40e9-932f-fe49a9c2885e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 13:31:12.876256 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.876212 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74d022bb-2968-40e9-932f-fe49a9c2885e" (UID: "74d022bb-2968-40e9-932f-fe49a9c2885e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:31:12.876256 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.876242 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74d022bb-2968-40e9-932f-fe49a9c2885e" (UID: "74d022bb-2968-40e9-932f-fe49a9c2885e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:31:12.877929 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.877901 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-api-access-4wwzx" (OuterVolumeSpecName: "kube-api-access-4wwzx") pod "74d022bb-2968-40e9-932f-fe49a9c2885e" (UID: "74d022bb-2968-40e9-932f-fe49a9c2885e"). InnerVolumeSpecName "kube-api-access-4wwzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 13:31:12.878028 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.877960 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d022bb-2968-40e9-932f-fe49a9c2885e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74d022bb-2968-40e9-932f-fe49a9c2885e" (UID: "74d022bb-2968-40e9-932f-fe49a9c2885e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:31:12.977114 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.977080 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:31:12.977114 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.977108 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wwzx\" (UniqueName: \"kubernetes.io/projected/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-api-access-4wwzx\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:31:12.977114 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.977119 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d022bb-2968-40e9-932f-fe49a9c2885e-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:31:12.977329 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.977129 2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d022bb-2968-40e9-932f-fe49a9c2885e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:31:12.977329 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:12.977138 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d022bb-2968-40e9-932f-fe49a9c2885e-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:31:13.220618 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.220527 2554 generic.go:356] "Generic (PLEG): container finished" podID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerID="1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c" exitCode=0 Sep 08 13:31:13.220618 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.220593 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" Sep 08 13:31:13.220618 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.220603 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" event={"ID":"74d022bb-2968-40e9-932f-fe49a9c2885e","Type":"ContainerDied","Data":"1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c"} Sep 08 13:31:13.220830 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.220662 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk" event={"ID":"74d022bb-2968-40e9-932f-fe49a9c2885e","Type":"ContainerDied","Data":"563597ea30fb9860badc9336822468230462d37d66967fcb8a003a2402196d2d"} Sep 08 13:31:13.220830 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.220716 2554 scope.go:117] "RemoveContainer" containerID="0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5" Sep 08 13:31:13.228919 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.228883 2554 scope.go:117] "RemoveContainer" containerID="1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c" Sep 08 13:31:13.235781 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.235762 2554 scope.go:117] "RemoveContainer" containerID="969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6" Sep 08 13:31:13.242395 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.242380 2554 scope.go:117] "RemoveContainer" containerID="0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5" Sep 08 13:31:13.242656 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:31:13.242634 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5\": container with ID starting with 0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5 not found: ID does not exist" containerID="0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5" Sep 08 13:31:13.242714 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.242664 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5"} err="failed to get container status \"0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5\": rpc error: code = NotFound desc = could not find container \"0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5\": container with ID starting with 0a43950092e977287f186a222c23cdbccf1e5171f259a1996570d8bab6a88ac5 not found: ID does not exist" Sep 08 13:31:13.242714 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.242696 2554 scope.go:117] "RemoveContainer" containerID="1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c" Sep 08 13:31:13.242967 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:31:13.242948 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c\": container with ID starting with 1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c not found: ID does not exist" containerID="1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c" Sep 08 13:31:13.243019 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.242972 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c"} err="failed to get container status \"1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c\": rpc error: code = NotFound desc = could not find container \"1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c\": container with ID starting with 1a0c55ad7ec6c43956b82c6b695d5131dbd314c8256f68d589a42f52627dcf3c not found: ID does not exist" Sep 08 13:31:13.243019 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.242988 2554 scope.go:117] "RemoveContainer" containerID="969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6" Sep 08 13:31:13.243187 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:31:13.243171 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6\": container with ID starting with 969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6 not found: ID does not exist" containerID="969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6" Sep 08 13:31:13.243226 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.243191 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6"} err="failed to get container status \"969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6\": rpc error: code = NotFound desc = could not find container \"969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6\": container with ID starting with 969fe4c0161c3d071f1a06729536ecea069e714fc2bb4568ff7657e0794ee0e6 not found: ID does not exist" Sep 08 13:31:13.249479 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.249457 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk"] Sep 08 13:31:13.257343 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:13.257323 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-766ff5986c-vkntk"] Sep 08 13:31:14.189754 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:14.189711 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" path="/var/lib/kubelet/pods/74d022bb-2968-40e9-932f-fe49a9c2885e/volumes" Sep 08 13:31:15.169129 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:15.169092 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:31:20.169507 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:20.169466 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:31:20.169927 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:20.169579 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:31:25.170065 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:25.170021 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:31:30.169946 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:30.169909 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:31:32.188205 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188172 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk"] Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188413 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188423 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188446 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="storage-initializer" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188451 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="storage-initializer" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188458 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188464 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188470 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kube-rbac-proxy" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188475 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kube-rbac-proxy" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188514 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f84bd86-43cc-4f30-b38a-180837aba088" containerName="switch-graph-f794f" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188522 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kube-rbac-proxy" Sep 08 13:31:32.188559 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.188529 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74d022bb-2968-40e9-932f-fe49a9c2885e" containerName="kserve-container" Sep 08 13:31:32.192726 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.192709 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:32.195240 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.195210 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-87be3-kube-rbac-proxy-sar-config\"" Sep 08 13:31:32.195240 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.195227 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-87be3-serving-cert\"" Sep 08 13:31:32.201041 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.201016 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk"] Sep 08 13:31:32.316930 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.316889 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73470095-ecbc-4f59-8821-21cd9ec14413-openshift-service-ca-bundle\") pod \"switch-graph-87be3-bdb646768-z4lgk\" (UID: \"73470095-ecbc-4f59-8821-21cd9ec14413\") " pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:32.316930 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.316938 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73470095-ecbc-4f59-8821-21cd9ec14413-proxy-tls\") pod \"switch-graph-87be3-bdb646768-z4lgk\" (UID: \"73470095-ecbc-4f59-8821-21cd9ec14413\") " pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:32.418314 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.418274 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73470095-ecbc-4f59-8821-21cd9ec14413-proxy-tls\") pod \"switch-graph-87be3-bdb646768-z4lgk\" (UID: \"73470095-ecbc-4f59-8821-21cd9ec14413\") " pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:32.418480 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.418348 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73470095-ecbc-4f59-8821-21cd9ec14413-openshift-service-ca-bundle\") pod \"switch-graph-87be3-bdb646768-z4lgk\" (UID: \"73470095-ecbc-4f59-8821-21cd9ec14413\") " pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:32.418957 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.418937 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73470095-ecbc-4f59-8821-21cd9ec14413-openshift-service-ca-bundle\") pod \"switch-graph-87be3-bdb646768-z4lgk\" (UID: \"73470095-ecbc-4f59-8821-21cd9ec14413\") " pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:32.420718 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.420693 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73470095-ecbc-4f59-8821-21cd9ec14413-proxy-tls\") pod \"switch-graph-87be3-bdb646768-z4lgk\" (UID: \"73470095-ecbc-4f59-8821-21cd9ec14413\") " pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:32.503361 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.503329 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:32.770385 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:32.770358 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk"] Sep 08 13:31:32.772750 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:31:32.772720 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73470095_ecbc_4f59_8821_21cd9ec14413.slice/crio-921e68cd42b8c4bf0b57bf46928eb9570f97e4059e45b40d51ef6374b0874eb7 WatchSource:0}: Error finding container 921e68cd42b8c4bf0b57bf46928eb9570f97e4059e45b40d51ef6374b0874eb7: Status 404 returned error can't find the container with id 921e68cd42b8c4bf0b57bf46928eb9570f97e4059e45b40d51ef6374b0874eb7 Sep 08 13:31:33.290943 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:33.290906 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" event={"ID":"73470095-ecbc-4f59-8821-21cd9ec14413","Type":"ContainerStarted","Data":"fa0e37d47abc3b3b9fb219d0aed4a8de88cd946a98bef122416e30017ef5f4ff"} Sep 08 13:31:33.291310 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:33.290959 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" event={"ID":"73470095-ecbc-4f59-8821-21cd9ec14413","Type":"ContainerStarted","Data":"921e68cd42b8c4bf0b57bf46928eb9570f97e4059e45b40d51ef6374b0874eb7"} Sep 08 13:31:33.291310 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:33.290991 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:33.311522 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:33.311471 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" podStartSLOduration=1.311456482 podStartE2EDuration="1.311456482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:31:33.311057845 +0000 UTC m=+919.643732494" watchObservedRunningTime="2026-09-08 13:31:33.311456482 +0000 UTC m=+919.644131119" Sep 08 13:31:35.169312 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:35.169275 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:31:38.030588 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.030563 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:31:38.160162 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.160065 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-proxy-tls\") pod \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\" (UID: \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\") " Sep 08 13:31:38.160162 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.160124 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-openshift-service-ca-bundle\") pod \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\" (UID: \"c281bc12-19c2-4af6-874d-0c23e4b1d9a1\") " Sep 08 13:31:38.160512 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.160489 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c281bc12-19c2-4af6-874d-0c23e4b1d9a1" (UID: "c281bc12-19c2-4af6-874d-0c23e4b1d9a1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:31:38.162123 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.162099 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c281bc12-19c2-4af6-874d-0c23e4b1d9a1" (UID: "c281bc12-19c2-4af6-874d-0c23e4b1d9a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:31:38.261647 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.261607 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:31:38.261647 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.261642 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c281bc12-19c2-4af6-874d-0c23e4b1d9a1-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:31:38.315616 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.315582 2554 generic.go:356] "Generic (PLEG): container finished" podID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerID="0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8" exitCode=0 Sep 08 13:31:38.315775 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.315627 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" Sep 08 13:31:38.315775 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.315626 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" event={"ID":"c281bc12-19c2-4af6-874d-0c23e4b1d9a1","Type":"ContainerDied","Data":"0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8"} Sep 08 13:31:38.315775 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.315695 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s" event={"ID":"c281bc12-19c2-4af6-874d-0c23e4b1d9a1","Type":"ContainerDied","Data":"5b677b92c96e46b6a919686debf453a9bc2318e29a829317a40e9c65987dbef5"} Sep 08 13:31:38.315775 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.315725 2554 scope.go:117] "RemoveContainer" containerID="0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8" Sep 08 13:31:38.323092 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.323063 2554 scope.go:117] "RemoveContainer" containerID="0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8" Sep 08 13:31:38.323395 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:31:38.323366 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8\": container with ID starting with 0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8 not found: ID does not exist" containerID="0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8" Sep 08 13:31:38.323471 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.323400 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8"} err="failed to get container status \"0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8\": rpc error: code = NotFound desc = could not find container \"0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8\": container with ID starting with 0ec050164ca74223c3730a0771315cdb757a0223a4812f2004d6b7a211ceecf8 not found: ID does not exist" Sep 08 13:31:38.333556 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.333532 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s"] Sep 08 13:31:38.339454 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:38.339428 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-546cb8556c-kwp6s"] Sep 08 13:31:39.299843 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:39.299815 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:31:40.187187 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:31:40.187152 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" path="/var/lib/kubelet/pods/c281bc12-19c2-4af6-874d-0c23e4b1d9a1/volumes" Sep 08 13:32:08.042204 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.042164 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4"] Sep 08 13:32:08.044709 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.042515 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" Sep 08 13:32:08.044709 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.042535 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" Sep 08 13:32:08.044709 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.042615 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c281bc12-19c2-4af6-874d-0c23e4b1d9a1" containerName="model-chainer" Sep 08 13:32:08.045594 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.045576 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:32:08.048193 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.048162 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-30f77-serving-cert\"" Sep 08 13:32:08.048193 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.048175 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-30f77-kube-rbac-proxy-sar-config\"" Sep 08 13:32:08.054090 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.054066 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4"] Sep 08 13:32:08.180141 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.180105 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81c6b279-d313-4eb4-ad16-c43cf5d4472d-proxy-tls\") pod \"sequence-graph-30f77-859fc5c779-w9th4\" (UID: \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\") " pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:32:08.180141 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.180142 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c6b279-d313-4eb4-ad16-c43cf5d4472d-openshift-service-ca-bundle\") pod \"sequence-graph-30f77-859fc5c779-w9th4\" (UID: \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\") " pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:32:08.280974 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.280936 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81c6b279-d313-4eb4-ad16-c43cf5d4472d-proxy-tls\") pod \"sequence-graph-30f77-859fc5c779-w9th4\" (UID: \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\") " pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:32:08.280974 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.280976 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c6b279-d313-4eb4-ad16-c43cf5d4472d-openshift-service-ca-bundle\") pod \"sequence-graph-30f77-859fc5c779-w9th4\" (UID: \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\") " pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:32:08.281576 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.281555 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c6b279-d313-4eb4-ad16-c43cf5d4472d-openshift-service-ca-bundle\") pod \"sequence-graph-30f77-859fc5c779-w9th4\" (UID: \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\") " pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:32:08.283459 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.283434 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81c6b279-d313-4eb4-ad16-c43cf5d4472d-proxy-tls\") pod \"sequence-graph-30f77-859fc5c779-w9th4\" (UID: \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\") " pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:32:08.356297 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.356217 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:32:08.624139 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:08.624051 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4"] Sep 08 13:32:08.627112 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:32:08.627083 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c6b279_d313_4eb4_ad16_c43cf5d4472d.slice/crio-d1416b2f67abb10ec1520e6675890a8bc5cd34e59ea8cd5d6595b312e826ff3d WatchSource:0}: Error finding container d1416b2f67abb10ec1520e6675890a8bc5cd34e59ea8cd5d6595b312e826ff3d: Status 404 returned error can't find the container with id d1416b2f67abb10ec1520e6675890a8bc5cd34e59ea8cd5d6595b312e826ff3d Sep 08 13:32:09.428209 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:09.428172 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" event={"ID":"81c6b279-d313-4eb4-ad16-c43cf5d4472d","Type":"ContainerStarted","Data":"9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608"} Sep 08 13:32:09.428629 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:09.428251 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" event={"ID":"81c6b279-d313-4eb4-ad16-c43cf5d4472d","Type":"ContainerStarted","Data":"d1416b2f67abb10ec1520e6675890a8bc5cd34e59ea8cd5d6595b312e826ff3d"} Sep 08 13:32:09.428629 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:09.428348 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:32:09.445877 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:09.445830 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" podStartSLOduration=1.445812916 podStartE2EDuration="1.445812916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:32:09.444286815 +0000 UTC m=+955.776961453" watchObservedRunningTime="2026-09-08 13:32:09.445812916 +0000 UTC m=+955.778487610" Sep 08 13:32:15.436313 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:32:15.436285 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:39:46.841742 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:39:46.841708 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk"] Sep 08 13:39:46.844097 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:39:46.841910 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" containerID="cri-o://fa0e37d47abc3b3b9fb219d0aed4a8de88cd946a98bef122416e30017ef5f4ff" gracePeriod=30 Sep 08 13:39:49.298705 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:39:49.298646 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:39:54.298264 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:39:54.298222 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:39:59.298839 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:39:59.298792 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:39:59.299224 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:39:59.298936 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:40:04.299022 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:04.298981 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:40:09.298623 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:09.298532 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:40:14.298418 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:14.298379 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:40:17.107918 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:17.107831 2554 generic.go:356] "Generic (PLEG): container finished" podID="73470095-ecbc-4f59-8821-21cd9ec14413" containerID="fa0e37d47abc3b3b9fb219d0aed4a8de88cd946a98bef122416e30017ef5f4ff" exitCode=0 Sep 08 13:40:17.107918 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:17.107880 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" event={"ID":"73470095-ecbc-4f59-8821-21cd9ec14413","Type":"ContainerDied","Data":"fa0e37d47abc3b3b9fb219d0aed4a8de88cd946a98bef122416e30017ef5f4ff"} Sep 08 13:40:17.480861 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:17.480833 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:40:17.545311 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:17.545270 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73470095-ecbc-4f59-8821-21cd9ec14413-proxy-tls\") pod \"73470095-ecbc-4f59-8821-21cd9ec14413\" (UID: \"73470095-ecbc-4f59-8821-21cd9ec14413\") " Sep 08 13:40:17.545311 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:17.545315 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73470095-ecbc-4f59-8821-21cd9ec14413-openshift-service-ca-bundle\") pod \"73470095-ecbc-4f59-8821-21cd9ec14413\" (UID: \"73470095-ecbc-4f59-8821-21cd9ec14413\") " Sep 08 13:40:17.545758 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:17.545733 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73470095-ecbc-4f59-8821-21cd9ec14413-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "73470095-ecbc-4f59-8821-21cd9ec14413" (UID: "73470095-ecbc-4f59-8821-21cd9ec14413"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:40:17.547392 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:17.547370 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73470095-ecbc-4f59-8821-21cd9ec14413-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73470095-ecbc-4f59-8821-21cd9ec14413" (UID: "73470095-ecbc-4f59-8821-21cd9ec14413"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:40:17.646440 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:17.646400 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73470095-ecbc-4f59-8821-21cd9ec14413-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:40:17.646440 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:17.646433 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73470095-ecbc-4f59-8821-21cd9ec14413-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:40:18.112652 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:18.112614 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" event={"ID":"73470095-ecbc-4f59-8821-21cd9ec14413","Type":"ContainerDied","Data":"921e68cd42b8c4bf0b57bf46928eb9570f97e4059e45b40d51ef6374b0874eb7"} Sep 08 13:40:18.112652 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:18.112641 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk" Sep 08 13:40:18.113251 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:18.112704 2554 scope.go:117] "RemoveContainer" containerID="fa0e37d47abc3b3b9fb219d0aed4a8de88cd946a98bef122416e30017ef5f4ff" Sep 08 13:40:18.136051 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:18.136018 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk"] Sep 08 13:40:18.144064 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:18.144041 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87be3-bdb646768-z4lgk"] Sep 08 13:40:18.187459 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:18.187425 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" path="/var/lib/kubelet/pods/73470095-ecbc-4f59-8821-21cd9ec14413/volumes" Sep 08 13:40:22.646667 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:22.646632 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4"] Sep 08 13:40:22.647112 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:22.646849 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" containerID="cri-o://9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608" gracePeriod=30 Sep 08 13:40:25.435133 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:25.435094 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:40:30.435691 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:30.435651 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:40:35.435065 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:35.435025 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:40:35.435520 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:35.435144 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:40:40.435345 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:40.435302 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:40:45.435277 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:45.435237 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:40:47.224730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.224694 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq"] Sep 08 13:40:47.225123 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.224944 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" Sep 08 13:40:47.225123 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.224954 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" Sep 08 13:40:47.225123 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.224996 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="73470095-ecbc-4f59-8821-21cd9ec14413" containerName="switch-graph-87be3" Sep 08 13:40:47.227809 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.227793 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:47.235543 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.235514 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-601fd-kube-rbac-proxy-sar-config\"" Sep 08 13:40:47.235543 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.235514 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-601fd-serving-cert\"" Sep 08 13:40:47.247022 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.246983 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq"] Sep 08 13:40:47.363634 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.363595 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-proxy-tls\") pod \"ensemble-graph-601fd-8f5c764c5-5vlqq\" (UID: \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\") " pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:47.363832 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.363647 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-openshift-service-ca-bundle\") pod \"ensemble-graph-601fd-8f5c764c5-5vlqq\" (UID: \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\") " pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:47.464615 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.464573 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-proxy-tls\") pod \"ensemble-graph-601fd-8f5c764c5-5vlqq\" (UID: \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\") " pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:47.464823 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.464633 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-openshift-service-ca-bundle\") pod \"ensemble-graph-601fd-8f5c764c5-5vlqq\" (UID: \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\") " pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:47.464823 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:40:47.464747 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-601fd-serving-cert: secret "ensemble-graph-601fd-serving-cert" not found Sep 08 13:40:47.464823 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:40:47.464817 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-proxy-tls podName:fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:40:47.964800242 +0000 UTC m=+1474.297474859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-proxy-tls") pod "ensemble-graph-601fd-8f5c764c5-5vlqq" (UID: "fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0") : secret "ensemble-graph-601fd-serving-cert" not found Sep 08 13:40:47.465254 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.465236 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-openshift-service-ca-bundle\") pod \"ensemble-graph-601fd-8f5c764c5-5vlqq\" (UID: \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\") " pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:47.969630 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.969587 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-proxy-tls\") pod \"ensemble-graph-601fd-8f5c764c5-5vlqq\" (UID: \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\") " pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:47.972102 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:47.972068 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-proxy-tls\") pod \"ensemble-graph-601fd-8f5c764c5-5vlqq\" (UID: \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\") " pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:48.136981 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:48.136938 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:48.415465 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:48.415431 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq"] Sep 08 13:40:48.417460 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:40:48.417434 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcc6425_d6ad_40ab_9fe1_4e3c1ada15f0.slice/crio-1bf76ddd5c6c7800861ae1550a5bc5e726999cf93f4ac4868ea36b8d943bf849 WatchSource:0}: Error finding container 1bf76ddd5c6c7800861ae1550a5bc5e726999cf93f4ac4868ea36b8d943bf849: Status 404 returned error can't find the container with id 1bf76ddd5c6c7800861ae1550a5bc5e726999cf93f4ac4868ea36b8d943bf849 Sep 08 13:40:49.225529 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:49.225492 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" event={"ID":"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0","Type":"ContainerStarted","Data":"7328c8756efbc731e809d2d3a62de1f77f0d64e2fd6a5752c2cb888de72dd89b"} Sep 08 13:40:49.225869 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:49.225560 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" event={"ID":"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0","Type":"ContainerStarted","Data":"1bf76ddd5c6c7800861ae1550a5bc5e726999cf93f4ac4868ea36b8d943bf849"} Sep 08 13:40:49.225869 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:49.225612 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:49.245371 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:49.245315 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" podStartSLOduration=2.245296412 podStartE2EDuration="2.245296412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:40:49.244773019 +0000 UTC m=+1475.577447685" watchObservedRunningTime="2026-09-08 13:40:49.245296412 +0000 UTC m=+1475.577971089" Sep 08 13:40:50.435296 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:50.435247 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:40:52.796522 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:52.796495 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:40:52.903824 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:52.903738 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c6b279-d313-4eb4-ad16-c43cf5d4472d-openshift-service-ca-bundle\") pod \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\" (UID: \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\") " Sep 08 13:40:52.903824 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:52.903805 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81c6b279-d313-4eb4-ad16-c43cf5d4472d-proxy-tls\") pod \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\" (UID: \"81c6b279-d313-4eb4-ad16-c43cf5d4472d\") " Sep 08 13:40:52.904110 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:52.904074 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c6b279-d313-4eb4-ad16-c43cf5d4472d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "81c6b279-d313-4eb4-ad16-c43cf5d4472d" (UID: "81c6b279-d313-4eb4-ad16-c43cf5d4472d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:40:52.905914 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:52.905891 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c6b279-d313-4eb4-ad16-c43cf5d4472d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81c6b279-d313-4eb4-ad16-c43cf5d4472d" (UID: "81c6b279-d313-4eb4-ad16-c43cf5d4472d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:40:53.004211 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.004168 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c6b279-d313-4eb4-ad16-c43cf5d4472d-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:40:53.004211 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.004204 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81c6b279-d313-4eb4-ad16-c43cf5d4472d-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:40:53.240060 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.239962 2554 generic.go:356] "Generic (PLEG): container finished" podID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerID="9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608" exitCode=0 Sep 08 13:40:53.240060 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.240012 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" Sep 08 13:40:53.240060 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.240042 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" event={"ID":"81c6b279-d313-4eb4-ad16-c43cf5d4472d","Type":"ContainerDied","Data":"9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608"} Sep 08 13:40:53.240343 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.240096 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4" event={"ID":"81c6b279-d313-4eb4-ad16-c43cf5d4472d","Type":"ContainerDied","Data":"d1416b2f67abb10ec1520e6675890a8bc5cd34e59ea8cd5d6595b312e826ff3d"} Sep 08 13:40:53.240343 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.240117 2554 scope.go:117] "RemoveContainer" containerID="9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608" Sep 08 13:40:53.248589 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.248569 2554 scope.go:117] "RemoveContainer" containerID="9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608" Sep 08 13:40:53.248925 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:40:53.248901 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608\": container with ID starting with 9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608 not found: ID does not exist" containerID="9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608" Sep 08 13:40:53.249000 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.248939 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608"} err="failed to get container status \"9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608\": rpc error: code = NotFound desc = could not find container \"9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608\": container with ID starting with 9f73878e7453df5ffcd3785894b461ba171fafca089a8e67aa049f71d8c0d608 not found: ID does not exist" Sep 08 13:40:53.264332 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.264299 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4"] Sep 08 13:40:53.269469 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:53.269444 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30f77-859fc5c779-w9th4"] Sep 08 13:40:54.187182 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:54.187148 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" path="/var/lib/kubelet/pods/81c6b279-d313-4eb4-ad16-c43cf5d4472d/volumes" Sep 08 13:40:55.234677 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:55.234644 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:40:57.250971 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:57.250940 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq"] Sep 08 13:40:57.251366 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:40:57.251120 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" containerID="cri-o://7328c8756efbc731e809d2d3a62de1f77f0d64e2fd6a5752c2cb888de72dd89b" gracePeriod=30 Sep 08 13:41:00.233641 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:00.233596 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:05.232782 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:05.232744 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:10.233905 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:10.233860 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:10.234365 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:10.233970 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:41:15.233283 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:15.233234 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:20.233611 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:20.233569 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:22.899010 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:22.898968 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2"] Sep 08 13:41:22.899567 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:22.899327 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" Sep 08 13:41:22.899567 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:22.899345 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" Sep 08 13:41:22.899567 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:22.899424 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c6b279-d313-4eb4-ad16-c43cf5d4472d" containerName="sequence-graph-30f77" Sep 08 13:41:22.902111 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:22.902090 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:22.904674 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:22.904654 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-e90df-kube-rbac-proxy-sar-config\"" Sep 08 13:41:22.904819 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:22.904801 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-e90df-serving-cert\"" Sep 08 13:41:22.915759 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:22.915731 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2"] Sep 08 13:41:23.022487 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:23.022444 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3edddb8-2e57-4e63-bb22-9736954c9204-proxy-tls\") pod \"sequence-graph-e90df-764845479d-h54j2\" (UID: \"b3edddb8-2e57-4e63-bb22-9736954c9204\") " pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:23.022657 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:23.022508 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3edddb8-2e57-4e63-bb22-9736954c9204-openshift-service-ca-bundle\") pod \"sequence-graph-e90df-764845479d-h54j2\" (UID: \"b3edddb8-2e57-4e63-bb22-9736954c9204\") " pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:23.123489 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:23.123450 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3edddb8-2e57-4e63-bb22-9736954c9204-proxy-tls\") pod \"sequence-graph-e90df-764845479d-h54j2\" (UID: \"b3edddb8-2e57-4e63-bb22-9736954c9204\") " pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:23.123648 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:23.123518 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3edddb8-2e57-4e63-bb22-9736954c9204-openshift-service-ca-bundle\") pod \"sequence-graph-e90df-764845479d-h54j2\" (UID: \"b3edddb8-2e57-4e63-bb22-9736954c9204\") " pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:23.123648 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:41:23.123610 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-e90df-serving-cert: secret "sequence-graph-e90df-serving-cert" not found Sep 08 13:41:23.123749 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:41:23.123702 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3edddb8-2e57-4e63-bb22-9736954c9204-proxy-tls podName:b3edddb8-2e57-4e63-bb22-9736954c9204 nodeName:}" failed. No retries permitted until 2026-09-08 13:41:23.623660852 +0000 UTC m=+1509.956335479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b3edddb8-2e57-4e63-bb22-9736954c9204-proxy-tls") pod "sequence-graph-e90df-764845479d-h54j2" (UID: "b3edddb8-2e57-4e63-bb22-9736954c9204") : secret "sequence-graph-e90df-serving-cert" not found Sep 08 13:41:23.124122 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:23.124101 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3edddb8-2e57-4e63-bb22-9736954c9204-openshift-service-ca-bundle\") pod \"sequence-graph-e90df-764845479d-h54j2\" (UID: \"b3edddb8-2e57-4e63-bb22-9736954c9204\") " pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:23.626726 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:23.626665 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3edddb8-2e57-4e63-bb22-9736954c9204-proxy-tls\") pod \"sequence-graph-e90df-764845479d-h54j2\" (UID: \"b3edddb8-2e57-4e63-bb22-9736954c9204\") " pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:23.629070 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:23.629049 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3edddb8-2e57-4e63-bb22-9736954c9204-proxy-tls\") pod \"sequence-graph-e90df-764845479d-h54j2\" (UID: \"b3edddb8-2e57-4e63-bb22-9736954c9204\") " pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:23.811218 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:23.811180 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:24.083829 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:24.083674 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2"] Sep 08 13:41:24.086394 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:41:24.086364 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3edddb8_2e57_4e63_bb22_9736954c9204.slice/crio-a3fc7a72fe3521e71f09b6026ccd323a93cd89d1e707bfbac801fd266b143b6e WatchSource:0}: Error finding container a3fc7a72fe3521e71f09b6026ccd323a93cd89d1e707bfbac801fd266b143b6e: Status 404 returned error can't find the container with id a3fc7a72fe3521e71f09b6026ccd323a93cd89d1e707bfbac801fd266b143b6e Sep 08 13:41:24.352403 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:24.352303 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" event={"ID":"b3edddb8-2e57-4e63-bb22-9736954c9204","Type":"ContainerStarted","Data":"e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835"} Sep 08 13:41:24.352403 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:24.352354 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" event={"ID":"b3edddb8-2e57-4e63-bb22-9736954c9204","Type":"ContainerStarted","Data":"a3fc7a72fe3521e71f09b6026ccd323a93cd89d1e707bfbac801fd266b143b6e"} Sep 08 13:41:24.352403 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:24.352382 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:24.370920 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:24.370871 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" podStartSLOduration=2.370856349 podStartE2EDuration="2.370856349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:24.370005213 +0000 UTC m=+1510.702679893" watchObservedRunningTime="2026-09-08 13:41:24.370856349 +0000 UTC m=+1510.703530986" Sep 08 13:41:25.234051 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:25.234010 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:27.363577 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:27.363545 2554 generic.go:356] "Generic (PLEG): container finished" podID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerID="7328c8756efbc731e809d2d3a62de1f77f0d64e2fd6a5752c2cb888de72dd89b" exitCode=0 Sep 08 13:41:27.363948 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:27.363619 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" event={"ID":"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0","Type":"ContainerDied","Data":"7328c8756efbc731e809d2d3a62de1f77f0d64e2fd6a5752c2cb888de72dd89b"} Sep 08 13:41:27.389518 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:27.389496 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:41:27.557771 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:27.557666 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-openshift-service-ca-bundle\") pod \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\" (UID: \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\") " Sep 08 13:41:27.557771 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:27.557725 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-proxy-tls\") pod \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\" (UID: \"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0\") " Sep 08 13:41:27.558023 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:27.558002 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" (UID: "fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:41:27.559717 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:27.559698 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" (UID: "fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:41:27.663449 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:27.659782 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:41:27.663449 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:27.659822 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:41:28.368734 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:28.368703 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" Sep 08 13:41:28.369215 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:28.368703 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq" event={"ID":"fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0","Type":"ContainerDied","Data":"1bf76ddd5c6c7800861ae1550a5bc5e726999cf93f4ac4868ea36b8d943bf849"} Sep 08 13:41:28.369215 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:28.368829 2554 scope.go:117] "RemoveContainer" containerID="7328c8756efbc731e809d2d3a62de1f77f0d64e2fd6a5752c2cb888de72dd89b" Sep 08 13:41:28.387031 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:28.387006 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq"] Sep 08 13:41:28.394869 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:28.394848 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-601fd-8f5c764c5-5vlqq"] Sep 08 13:41:30.187090 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:30.187052 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" path="/var/lib/kubelet/pods/fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0/volumes" Sep 08 13:41:30.361592 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:30.361566 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:32.995282 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:32.995249 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2"] Sep 08 13:41:32.995714 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:32.995427 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" containerID="cri-o://e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835" gracePeriod=30 Sep 08 13:41:35.360068 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:35.360024 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:40.360535 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:40.360441 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:45.359827 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:45.359789 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:45.360218 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:45.359932 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:41:50.359846 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:50.359810 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:55.360040 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:55.359999 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:41:57.495234 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.495196 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b"] Sep 08 13:41:57.495652 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.495457 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" Sep 08 13:41:57.495652 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.495468 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" Sep 08 13:41:57.495652 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.495508 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdcc6425-d6ad-40ab-9fe1-4e3c1ada15f0" containerName="ensemble-graph-601fd" Sep 08 13:41:57.498433 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.498410 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:57.500942 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.500915 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-64898-serving-cert\"" Sep 08 13:41:57.501058 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.500937 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-64898-kube-rbac-proxy-sar-config\"" Sep 08 13:41:57.506638 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.506614 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b"] Sep 08 13:41:57.567695 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.567651 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d2e82-76f9-4b33-8a9e-981f78846597-openshift-service-ca-bundle\") pod \"ensemble-graph-64898-76dcdb6787-r4g2b\" (UID: \"ea6d2e82-76f9-4b33-8a9e-981f78846597\") " pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:57.567820 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.567702 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea6d2e82-76f9-4b33-8a9e-981f78846597-proxy-tls\") pod \"ensemble-graph-64898-76dcdb6787-r4g2b\" (UID: \"ea6d2e82-76f9-4b33-8a9e-981f78846597\") " pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:57.668372 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.668341 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d2e82-76f9-4b33-8a9e-981f78846597-openshift-service-ca-bundle\") pod \"ensemble-graph-64898-76dcdb6787-r4g2b\" (UID: \"ea6d2e82-76f9-4b33-8a9e-981f78846597\") " pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:57.668514 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.668382 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea6d2e82-76f9-4b33-8a9e-981f78846597-proxy-tls\") pod \"ensemble-graph-64898-76dcdb6787-r4g2b\" (UID: \"ea6d2e82-76f9-4b33-8a9e-981f78846597\") " pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:57.668514 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:41:57.668489 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-64898-serving-cert: secret "ensemble-graph-64898-serving-cert" not found Sep 08 13:41:57.668583 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:41:57.668548 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea6d2e82-76f9-4b33-8a9e-981f78846597-proxy-tls podName:ea6d2e82-76f9-4b33-8a9e-981f78846597 nodeName:}" failed. No retries permitted until 2026-09-08 13:41:58.168532958 +0000 UTC m=+1544.501207574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ea6d2e82-76f9-4b33-8a9e-981f78846597-proxy-tls") pod "ensemble-graph-64898-76dcdb6787-r4g2b" (UID: "ea6d2e82-76f9-4b33-8a9e-981f78846597") : secret "ensemble-graph-64898-serving-cert" not found Sep 08 13:41:57.668991 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:57.668972 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d2e82-76f9-4b33-8a9e-981f78846597-openshift-service-ca-bundle\") pod \"ensemble-graph-64898-76dcdb6787-r4g2b\" (UID: \"ea6d2e82-76f9-4b33-8a9e-981f78846597\") " pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:58.171978 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:58.171946 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea6d2e82-76f9-4b33-8a9e-981f78846597-proxy-tls\") pod \"ensemble-graph-64898-76dcdb6787-r4g2b\" (UID: \"ea6d2e82-76f9-4b33-8a9e-981f78846597\") " pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:58.174274 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:58.174253 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea6d2e82-76f9-4b33-8a9e-981f78846597-proxy-tls\") pod \"ensemble-graph-64898-76dcdb6787-r4g2b\" (UID: \"ea6d2e82-76f9-4b33-8a9e-981f78846597\") " pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:58.408801 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:58.408770 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:58.678184 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:58.678151 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b"] Sep 08 13:41:58.680911 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:41:58.680885 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6d2e82_76f9_4b33_8a9e_981f78846597.slice/crio-6f650ab11a154d3c0c8cf5fd98af43dc2d56ab2d7ab293019fee9b7086a0792f WatchSource:0}: Error finding container 6f650ab11a154d3c0c8cf5fd98af43dc2d56ab2d7ab293019fee9b7086a0792f: Status 404 returned error can't find the container with id 6f650ab11a154d3c0c8cf5fd98af43dc2d56ab2d7ab293019fee9b7086a0792f Sep 08 13:41:59.476833 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:59.476797 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" event={"ID":"ea6d2e82-76f9-4b33-8a9e-981f78846597","Type":"ContainerStarted","Data":"b0212c16fcd6ff2b761823972ade7fff895b758b194b10f6f29a88ac2182ae7d"} Sep 08 13:41:59.477078 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:59.476848 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" event={"ID":"ea6d2e82-76f9-4b33-8a9e-981f78846597","Type":"ContainerStarted","Data":"6f650ab11a154d3c0c8cf5fd98af43dc2d56ab2d7ab293019fee9b7086a0792f"} Sep 08 13:41:59.477078 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:59.476878 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:41:59.495026 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:41:59.494978 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" podStartSLOduration=2.494962049 podStartE2EDuration="2.494962049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:59.494611559 +0000 UTC m=+1545.827286210" watchObservedRunningTime="2026-09-08 13:41:59.494962049 +0000 UTC m=+1545.827636686" Sep 08 13:42:00.360484 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:00.360448 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:42:03.135850 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.135820 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:42:03.208730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.208698 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3edddb8-2e57-4e63-bb22-9736954c9204-proxy-tls\") pod \"b3edddb8-2e57-4e63-bb22-9736954c9204\" (UID: \"b3edddb8-2e57-4e63-bb22-9736954c9204\") " Sep 08 13:42:03.208879 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.208750 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3edddb8-2e57-4e63-bb22-9736954c9204-openshift-service-ca-bundle\") pod \"b3edddb8-2e57-4e63-bb22-9736954c9204\" (UID: \"b3edddb8-2e57-4e63-bb22-9736954c9204\") " Sep 08 13:42:03.209146 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.209120 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3edddb8-2e57-4e63-bb22-9736954c9204-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b3edddb8-2e57-4e63-bb22-9736954c9204" (UID: "b3edddb8-2e57-4e63-bb22-9736954c9204"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:42:03.210833 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.210813 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3edddb8-2e57-4e63-bb22-9736954c9204-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3edddb8-2e57-4e63-bb22-9736954c9204" (UID: "b3edddb8-2e57-4e63-bb22-9736954c9204"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:42:03.309745 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.309657 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3edddb8-2e57-4e63-bb22-9736954c9204-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:42:03.309745 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.309713 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3edddb8-2e57-4e63-bb22-9736954c9204-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:42:03.492570 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.492536 2554 generic.go:356] "Generic (PLEG): container finished" podID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerID="e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835" exitCode=0 Sep 08 13:42:03.492752 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.492579 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" Sep 08 13:42:03.492752 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.492601 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" event={"ID":"b3edddb8-2e57-4e63-bb22-9736954c9204","Type":"ContainerDied","Data":"e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835"} Sep 08 13:42:03.492752 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.492645 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2" event={"ID":"b3edddb8-2e57-4e63-bb22-9736954c9204","Type":"ContainerDied","Data":"a3fc7a72fe3521e71f09b6026ccd323a93cd89d1e707bfbac801fd266b143b6e"} Sep 08 13:42:03.492752 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.492667 2554 scope.go:117] "RemoveContainer" containerID="e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835" Sep 08 13:42:03.501866 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.501845 2554 scope.go:117] "RemoveContainer" containerID="e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835" Sep 08 13:42:03.502130 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:42:03.502112 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835\": container with ID starting with e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835 not found: ID does not exist" containerID="e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835" Sep 08 13:42:03.502180 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.502139 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835"} err="failed to get container status \"e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835\": rpc error: code = NotFound desc = could not find container \"e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835\": container with ID starting with e09c17da3e0bc9618dfb45dd886cdee7c22ab1e4d1367e743ecc978d4c25e835 not found: ID does not exist" Sep 08 13:42:03.515663 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.515632 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2"] Sep 08 13:42:03.520254 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:03.520233 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e90df-764845479d-h54j2"] Sep 08 13:42:04.187669 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:04.187636 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" path="/var/lib/kubelet/pods/b3edddb8-2e57-4e63-bb22-9736954c9204/volumes" Sep 08 13:42:05.487211 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:05.487177 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:42:33.257355 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.257316 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt"] Sep 08 13:42:33.257856 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.257575 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" Sep 08 13:42:33.257856 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.257587 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" Sep 08 13:42:33.257856 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.257636 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3edddb8-2e57-4e63-bb22-9736954c9204" containerName="sequence-graph-e90df" Sep 08 13:42:33.260513 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.260494 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:33.262900 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.262876 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c7da0-serving-cert\"" Sep 08 13:42:33.263011 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.262985 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c7da0-kube-rbac-proxy-sar-config\"" Sep 08 13:42:33.271035 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.271016 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt"] Sep 08 13:42:33.326522 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.326483 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-proxy-tls\") pod \"sequence-graph-c7da0-7b7778f46-tqcxt\" (UID: \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\") " pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:33.326723 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.326535 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-openshift-service-ca-bundle\") pod \"sequence-graph-c7da0-7b7778f46-tqcxt\" (UID: \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\") " pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:33.427592 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.427544 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-openshift-service-ca-bundle\") pod \"sequence-graph-c7da0-7b7778f46-tqcxt\" (UID: \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\") " pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:33.427826 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.427639 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-proxy-tls\") pod \"sequence-graph-c7da0-7b7778f46-tqcxt\" (UID: \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\") " pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:33.427826 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:42:33.427784 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-c7da0-serving-cert: secret "sequence-graph-c7da0-serving-cert" not found Sep 08 13:42:33.427952 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:42:33.427868 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-proxy-tls podName:0d540e90-6dc4-4bc4-b048-e6205e12d0fc nodeName:}" failed. No retries permitted until 2026-09-08 13:42:33.927844249 +0000 UTC m=+1580.260518865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-proxy-tls") pod "sequence-graph-c7da0-7b7778f46-tqcxt" (UID: "0d540e90-6dc4-4bc4-b048-e6205e12d0fc") : secret "sequence-graph-c7da0-serving-cert" not found Sep 08 13:42:33.428207 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.428185 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-openshift-service-ca-bundle\") pod \"sequence-graph-c7da0-7b7778f46-tqcxt\" (UID: \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\") " pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:33.932285 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.932246 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-proxy-tls\") pod \"sequence-graph-c7da0-7b7778f46-tqcxt\" (UID: \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\") " pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:33.934625 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:33.934603 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-proxy-tls\") pod \"sequence-graph-c7da0-7b7778f46-tqcxt\" (UID: \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\") " pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:34.169885 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:34.169848 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:34.451136 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:34.451102 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt"] Sep 08 13:42:34.454075 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:42:34.454050 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d540e90_6dc4_4bc4_b048_e6205e12d0fc.slice/crio-4de53cb4b981daa84a83df3345c82d19663a2366527938b26031597ce7ece29c WatchSource:0}: Error finding container 4de53cb4b981daa84a83df3345c82d19663a2366527938b26031597ce7ece29c: Status 404 returned error can't find the container with id 4de53cb4b981daa84a83df3345c82d19663a2366527938b26031597ce7ece29c Sep 08 13:42:34.599995 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:34.599957 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" event={"ID":"0d540e90-6dc4-4bc4-b048-e6205e12d0fc","Type":"ContainerStarted","Data":"3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794"} Sep 08 13:42:34.600198 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:34.600007 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" event={"ID":"0d540e90-6dc4-4bc4-b048-e6205e12d0fc","Type":"ContainerStarted","Data":"4de53cb4b981daa84a83df3345c82d19663a2366527938b26031597ce7ece29c"} Sep 08 13:42:34.600198 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:34.600091 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:42:34.618038 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:34.617990 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" podStartSLOduration=1.617973021 podStartE2EDuration="1.617973021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:42:34.616657674 +0000 UTC m=+1580.949332314" watchObservedRunningTime="2026-09-08 13:42:34.617973021 +0000 UTC m=+1580.950647691" Sep 08 13:42:40.610080 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:42:40.610029 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:50:12.155170 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:12.155131 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b"] Sep 08 13:50:12.155673 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:12.155342 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" containerID="cri-o://b0212c16fcd6ff2b761823972ade7fff895b758b194b10f6f29a88ac2182ae7d" gracePeriod=30 Sep 08 13:50:15.485440 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:15.485391 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:50:20.485732 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:20.485666 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:50:25.486094 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:25.486033 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:50:25.486620 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:25.486196 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:50:30.485742 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:30.485691 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:50:35.486053 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:35.486009 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:50:40.485910 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:40.485820 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:50:42.284970 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:42.284938 2554 generic.go:356] "Generic (PLEG): container finished" podID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerID="b0212c16fcd6ff2b761823972ade7fff895b758b194b10f6f29a88ac2182ae7d" exitCode=0 Sep 08 13:50:42.285256 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:42.285004 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" event={"ID":"ea6d2e82-76f9-4b33-8a9e-981f78846597","Type":"ContainerDied","Data":"b0212c16fcd6ff2b761823972ade7fff895b758b194b10f6f29a88ac2182ae7d"} Sep 08 13:50:42.300028 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:42.300003 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:50:42.375207 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:42.375166 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d2e82-76f9-4b33-8a9e-981f78846597-openshift-service-ca-bundle\") pod \"ea6d2e82-76f9-4b33-8a9e-981f78846597\" (UID: \"ea6d2e82-76f9-4b33-8a9e-981f78846597\") " Sep 08 13:50:42.375207 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:42.375211 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea6d2e82-76f9-4b33-8a9e-981f78846597-proxy-tls\") pod \"ea6d2e82-76f9-4b33-8a9e-981f78846597\" (UID: \"ea6d2e82-76f9-4b33-8a9e-981f78846597\") " Sep 08 13:50:42.375540 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:42.375516 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6d2e82-76f9-4b33-8a9e-981f78846597-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ea6d2e82-76f9-4b33-8a9e-981f78846597" (UID: "ea6d2e82-76f9-4b33-8a9e-981f78846597"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:50:42.377271 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:42.377249 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea6d2e82-76f9-4b33-8a9e-981f78846597-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea6d2e82-76f9-4b33-8a9e-981f78846597" (UID: "ea6d2e82-76f9-4b33-8a9e-981f78846597"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:50:42.475929 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:42.475839 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d2e82-76f9-4b33-8a9e-981f78846597-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:50:42.475929 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:42.475875 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea6d2e82-76f9-4b33-8a9e-981f78846597-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:50:43.289511 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:43.289478 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" event={"ID":"ea6d2e82-76f9-4b33-8a9e-981f78846597","Type":"ContainerDied","Data":"6f650ab11a154d3c0c8cf5fd98af43dc2d56ab2d7ab293019fee9b7086a0792f"} Sep 08 13:50:43.289511 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:43.289503 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b" Sep 08 13:50:43.290012 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:43.289537 2554 scope.go:117] "RemoveContainer" containerID="b0212c16fcd6ff2b761823972ade7fff895b758b194b10f6f29a88ac2182ae7d" Sep 08 13:50:43.311121 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:43.311093 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b"] Sep 08 13:50:43.315710 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:43.315673 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64898-76dcdb6787-r4g2b"] Sep 08 13:50:44.187908 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:44.187869 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" path="/var/lib/kubelet/pods/ea6d2e82-76f9-4b33-8a9e-981f78846597/volumes" Sep 08 13:50:47.708530 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:47.708494 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt"] Sep 08 13:50:47.708950 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:47.708733 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" containerID="cri-o://3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794" gracePeriod=30 Sep 08 13:50:50.608580 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:50.608537 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:50:55.608057 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:50:55.608015 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:00.608373 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:00.608333 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:00.608856 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:00.608450 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:51:05.608198 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:05.608157 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:10.608576 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:10.608534 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:12.460510 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.460472 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4"] Sep 08 13:51:12.460943 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.460771 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" Sep 08 13:51:12.460943 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.460786 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" Sep 08 13:51:12.460943 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.460840 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea6d2e82-76f9-4b33-8a9e-981f78846597" containerName="ensemble-graph-64898" Sep 08 13:51:12.463673 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.463657 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:12.467025 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.467006 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-13b40-kube-rbac-proxy-sar-config\"" Sep 08 13:51:12.467509 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.467488 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-13b40-serving-cert\"" Sep 08 13:51:12.476882 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.476858 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4"] Sep 08 13:51:12.496736 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.496702 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0553c5d9-6a94-4409-a358-32bac469b315-proxy-tls\") pod \"splitter-graph-13b40-64fb84556f-s67d4\" (UID: \"0553c5d9-6a94-4409-a358-32bac469b315\") " pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:12.496736 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.496737 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0553c5d9-6a94-4409-a358-32bac469b315-openshift-service-ca-bundle\") pod \"splitter-graph-13b40-64fb84556f-s67d4\" (UID: \"0553c5d9-6a94-4409-a358-32bac469b315\") " pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:12.597358 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.597318 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0553c5d9-6a94-4409-a358-32bac469b315-proxy-tls\") pod \"splitter-graph-13b40-64fb84556f-s67d4\" (UID: \"0553c5d9-6a94-4409-a358-32bac469b315\") " pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:12.597358 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.597362 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0553c5d9-6a94-4409-a358-32bac469b315-openshift-service-ca-bundle\") pod \"splitter-graph-13b40-64fb84556f-s67d4\" (UID: \"0553c5d9-6a94-4409-a358-32bac469b315\") " pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:12.597569 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:51:12.597465 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-13b40-serving-cert: secret "splitter-graph-13b40-serving-cert" not found Sep 08 13:51:12.597569 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:51:12.597554 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0553c5d9-6a94-4409-a358-32bac469b315-proxy-tls podName:0553c5d9-6a94-4409-a358-32bac469b315 nodeName:}" failed. No retries permitted until 2026-09-08 13:51:13.097531272 +0000 UTC m=+2099.430205888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0553c5d9-6a94-4409-a358-32bac469b315-proxy-tls") pod "splitter-graph-13b40-64fb84556f-s67d4" (UID: "0553c5d9-6a94-4409-a358-32bac469b315") : secret "splitter-graph-13b40-serving-cert" not found Sep 08 13:51:12.598057 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:12.598038 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0553c5d9-6a94-4409-a358-32bac469b315-openshift-service-ca-bundle\") pod \"splitter-graph-13b40-64fb84556f-s67d4\" (UID: \"0553c5d9-6a94-4409-a358-32bac469b315\") " pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:13.101836 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:13.101781 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0553c5d9-6a94-4409-a358-32bac469b315-proxy-tls\") pod \"splitter-graph-13b40-64fb84556f-s67d4\" (UID: \"0553c5d9-6a94-4409-a358-32bac469b315\") " pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:13.104158 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:13.104138 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0553c5d9-6a94-4409-a358-32bac469b315-proxy-tls\") pod \"splitter-graph-13b40-64fb84556f-s67d4\" (UID: \"0553c5d9-6a94-4409-a358-32bac469b315\") " pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:13.373803 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:13.373654 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:13.656243 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:13.656166 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4"] Sep 08 13:51:13.658961 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:51:13.658925 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0553c5d9_6a94_4409_a358_32bac469b315.slice/crio-a61ec3a47bd4b73f2fe19a5a164f9878eb350de5186ef5707b4a4f9d4bceca96 WatchSource:0}: Error finding container a61ec3a47bd4b73f2fe19a5a164f9878eb350de5186ef5707b4a4f9d4bceca96: Status 404 returned error can't find the container with id a61ec3a47bd4b73f2fe19a5a164f9878eb350de5186ef5707b4a4f9d4bceca96 Sep 08 13:51:14.398716 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:14.398662 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" event={"ID":"0553c5d9-6a94-4409-a358-32bac469b315","Type":"ContainerStarted","Data":"5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b"} Sep 08 13:51:14.398886 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:14.398733 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" event={"ID":"0553c5d9-6a94-4409-a358-32bac469b315","Type":"ContainerStarted","Data":"a61ec3a47bd4b73f2fe19a5a164f9878eb350de5186ef5707b4a4f9d4bceca96"} Sep 08 13:51:14.398886 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:14.398767 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:14.425623 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:14.425541 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" podStartSLOduration=2.4255232700000002 podStartE2EDuration="2.42552327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:51:14.424467832 +0000 UTC m=+2100.757142469" watchObservedRunningTime="2026-09-08 13:51:14.42552327 +0000 UTC m=+2100.758197907" Sep 08 13:51:15.608346 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:15.608305 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:17.888121 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:17.888092 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:51:17.936413 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:17.936348 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-openshift-service-ca-bundle\") pod \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\" (UID: \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\") " Sep 08 13:51:17.936413 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:17.936417 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-proxy-tls\") pod \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\" (UID: \"0d540e90-6dc4-4bc4-b048-e6205e12d0fc\") " Sep 08 13:51:17.936797 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:17.936770 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0d540e90-6dc4-4bc4-b048-e6205e12d0fc" (UID: "0d540e90-6dc4-4bc4-b048-e6205e12d0fc"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:51:17.938494 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:17.938472 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d540e90-6dc4-4bc4-b048-e6205e12d0fc" (UID: "0d540e90-6dc4-4bc4-b048-e6205e12d0fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:51:18.037626 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.037576 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:51:18.037626 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.037624 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d540e90-6dc4-4bc4-b048-e6205e12d0fc-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:51:18.414485 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.414391 2554 generic.go:356] "Generic (PLEG): container finished" podID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerID="3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794" exitCode=0 Sep 08 13:51:18.414485 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.414443 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" Sep 08 13:51:18.414730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.414482 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" event={"ID":"0d540e90-6dc4-4bc4-b048-e6205e12d0fc","Type":"ContainerDied","Data":"3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794"} Sep 08 13:51:18.414730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.414538 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt" event={"ID":"0d540e90-6dc4-4bc4-b048-e6205e12d0fc","Type":"ContainerDied","Data":"4de53cb4b981daa84a83df3345c82d19663a2366527938b26031597ce7ece29c"} Sep 08 13:51:18.414730 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.414563 2554 scope.go:117] "RemoveContainer" containerID="3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794" Sep 08 13:51:18.422373 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.422355 2554 scope.go:117] "RemoveContainer" containerID="3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794" Sep 08 13:51:18.422668 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:51:18.422646 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794\": container with ID starting with 3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794 not found: ID does not exist" containerID="3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794" Sep 08 13:51:18.422726 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.422693 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794"} err="failed to get container status \"3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794\": rpc error: code = NotFound desc = could not find container \"3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794\": container with ID starting with 3d87f7d75efef4011f955d6a6c45c08b6db3e4f4b2db73b24db9d0dfa19e4794 not found: ID does not exist" Sep 08 13:51:18.444621 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.444588 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt"] Sep 08 13:51:18.454653 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:18.454617 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c7da0-7b7778f46-tqcxt"] Sep 08 13:51:20.187602 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:20.187567 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" path="/var/lib/kubelet/pods/0d540e90-6dc4-4bc4-b048-e6205e12d0fc/volumes" Sep 08 13:51:20.408165 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:20.408138 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:22.520504 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:22.520465 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4"] Sep 08 13:51:22.520915 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:22.520729 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" containerID="cri-o://5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b" gracePeriod=30 Sep 08 13:51:25.406997 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:25.406960 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:30.406539 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:30.406485 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:35.406613 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:35.406571 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:35.407170 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:35.406717 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:40.406342 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:40.406295 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:45.406277 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:45.406237 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:47.960161 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:47.960126 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj"] Sep 08 13:51:47.960589 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:47.960397 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" Sep 08 13:51:47.960589 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:47.960408 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" Sep 08 13:51:47.960589 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:47.960460 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d540e90-6dc4-4bc4-b048-e6205e12d0fc" containerName="sequence-graph-c7da0" Sep 08 13:51:47.964652 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:47.964633 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:51:47.968520 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:47.968501 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ce1f6-serving-cert\"" Sep 08 13:51:47.968933 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:47.968918 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ce1f6-kube-rbac-proxy-sar-config\"" Sep 08 13:51:47.979610 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:47.979589 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj"] Sep 08 13:51:48.061565 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:48.061518 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4297ef51-ddc1-4f75-bc84-1dd4e804854b-proxy-tls\") pod \"switch-graph-ce1f6-75dbfb64bd-q4pxj\" (UID: \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\") " pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:51:48.061565 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:48.061565 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4297ef51-ddc1-4f75-bc84-1dd4e804854b-openshift-service-ca-bundle\") pod \"switch-graph-ce1f6-75dbfb64bd-q4pxj\" (UID: \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\") " pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:51:48.162492 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:48.162457 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4297ef51-ddc1-4f75-bc84-1dd4e804854b-proxy-tls\") pod \"switch-graph-ce1f6-75dbfb64bd-q4pxj\" (UID: \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\") " pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:51:48.162670 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:48.162499 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4297ef51-ddc1-4f75-bc84-1dd4e804854b-openshift-service-ca-bundle\") pod \"switch-graph-ce1f6-75dbfb64bd-q4pxj\" (UID: \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\") " pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:51:48.163172 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:48.163146 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4297ef51-ddc1-4f75-bc84-1dd4e804854b-openshift-service-ca-bundle\") pod \"switch-graph-ce1f6-75dbfb64bd-q4pxj\" (UID: \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\") " pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:51:48.164914 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:48.164889 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4297ef51-ddc1-4f75-bc84-1dd4e804854b-proxy-tls\") pod \"switch-graph-ce1f6-75dbfb64bd-q4pxj\" (UID: \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\") " pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:51:48.274353 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:48.274308 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:51:48.547129 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:48.547040 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj"] Sep 08 13:51:48.550879 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:51:48.550851 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4297ef51_ddc1_4f75_bc84_1dd4e804854b.slice/crio-a276f38b0f1156ae481e0605f726efa3858a1506433d6f242772d19160f35d95 WatchSource:0}: Error finding container a276f38b0f1156ae481e0605f726efa3858a1506433d6f242772d19160f35d95: Status 404 returned error can't find the container with id a276f38b0f1156ae481e0605f726efa3858a1506433d6f242772d19160f35d95 Sep 08 13:51:49.520973 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:49.520939 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" event={"ID":"4297ef51-ddc1-4f75-bc84-1dd4e804854b","Type":"ContainerStarted","Data":"5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6"} Sep 08 13:51:49.521331 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:49.520991 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" event={"ID":"4297ef51-ddc1-4f75-bc84-1dd4e804854b","Type":"ContainerStarted","Data":"a276f38b0f1156ae481e0605f726efa3858a1506433d6f242772d19160f35d95"} Sep 08 13:51:49.521331 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:49.521059 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:51:49.543429 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:49.543358 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" podStartSLOduration=2.543342816 podStartE2EDuration="2.543342816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:51:49.543125916 +0000 UTC m=+2135.875800552" watchObservedRunningTime="2026-09-08 13:51:49.543342816 +0000 UTC m=+2135.876017463" Sep 08 13:51:50.406162 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:50.406126 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 13:51:52.658447 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:52.658424 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:52.798147 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:52.798058 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0553c5d9-6a94-4409-a358-32bac469b315-openshift-service-ca-bundle\") pod \"0553c5d9-6a94-4409-a358-32bac469b315\" (UID: \"0553c5d9-6a94-4409-a358-32bac469b315\") " Sep 08 13:51:52.798147 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:52.798114 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0553c5d9-6a94-4409-a358-32bac469b315-proxy-tls\") pod \"0553c5d9-6a94-4409-a358-32bac469b315\" (UID: \"0553c5d9-6a94-4409-a358-32bac469b315\") " Sep 08 13:51:52.798462 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:52.798442 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0553c5d9-6a94-4409-a358-32bac469b315-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0553c5d9-6a94-4409-a358-32bac469b315" (UID: "0553c5d9-6a94-4409-a358-32bac469b315"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 13:51:52.800144 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:52.800118 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0553c5d9-6a94-4409-a358-32bac469b315-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0553c5d9-6a94-4409-a358-32bac469b315" (UID: "0553c5d9-6a94-4409-a358-32bac469b315"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 13:51:52.899477 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:52.899423 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0553c5d9-6a94-4409-a358-32bac469b315-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:51:52.899477 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:52.899466 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0553c5d9-6a94-4409-a358-32bac469b315-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 13:51:53.535921 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:53.535886 2554 generic.go:356] "Generic (PLEG): container finished" podID="0553c5d9-6a94-4409-a358-32bac469b315" containerID="5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b" exitCode=0 Sep 08 13:51:53.536090 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:53.535934 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" Sep 08 13:51:53.536090 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:53.535946 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" event={"ID":"0553c5d9-6a94-4409-a358-32bac469b315","Type":"ContainerDied","Data":"5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b"} Sep 08 13:51:53.536090 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:53.535991 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4" event={"ID":"0553c5d9-6a94-4409-a358-32bac469b315","Type":"ContainerDied","Data":"a61ec3a47bd4b73f2fe19a5a164f9878eb350de5186ef5707b4a4f9d4bceca96"} Sep 08 13:51:53.536090 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:53.536019 2554 scope.go:117] "RemoveContainer" containerID="5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b" Sep 08 13:51:53.548552 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:53.548533 2554 scope.go:117] "RemoveContainer" containerID="5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b" Sep 08 13:51:53.548852 ip-10-0-133-71 kubenswrapper[2554]: E0908 13:51:53.548832 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b\": container with ID starting with 5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b not found: ID does not exist" containerID="5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b" Sep 08 13:51:53.548917 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:53.548861 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b"} err="failed to get container status \"5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b\": rpc error: code = NotFound desc = could not find container \"5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b\": container with ID starting with 5297b9040cc9016c2a30bd16cd4d0617f58c62db1f9972dd7a3ba28326ad6f3b not found: ID does not exist" Sep 08 13:51:53.560234 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:53.560211 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4"] Sep 08 13:51:53.563867 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:53.563847 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-13b40-64fb84556f-s67d4"] Sep 08 13:51:54.187103 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:54.187068 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0553c5d9-6a94-4409-a358-32bac469b315" path="/var/lib/kubelet/pods/0553c5d9-6a94-4409-a358-32bac469b315/volumes" Sep 08 13:51:55.530519 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:51:55.530491 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 13:52:22.792435 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.792400 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w"] Sep 08 13:52:22.792912 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.792643 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" Sep 08 13:52:22.792912 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.792653 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" Sep 08 13:52:22.792912 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.792736 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0553c5d9-6a94-4409-a358-32bac469b315" containerName="splitter-graph-13b40" Sep 08 13:52:22.795534 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.795515 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 13:52:22.798801 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.798762 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7d014-serving-cert\"" Sep 08 13:52:22.799178 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.799162 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7d014-kube-rbac-proxy-sar-config\"" Sep 08 13:52:22.804979 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.804956 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w"] Sep 08 13:52:22.916312 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.916250 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-openshift-service-ca-bundle\") pod \"splitter-graph-7d014-7f8d646646-jnp9w\" (UID: \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\") " pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 13:52:22.916312 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:22.916310 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-proxy-tls\") pod \"splitter-graph-7d014-7f8d646646-jnp9w\" (UID: \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\") " pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 13:52:23.017187 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.017137 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-openshift-service-ca-bundle\") pod \"splitter-graph-7d014-7f8d646646-jnp9w\" (UID: \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\") " pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 13:52:23.017187 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.017191 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-proxy-tls\") pod \"splitter-graph-7d014-7f8d646646-jnp9w\" (UID: \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\") " pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 13:52:23.018028 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.017989 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-openshift-service-ca-bundle\") pod \"splitter-graph-7d014-7f8d646646-jnp9w\" (UID: \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\") " pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 13:52:23.020440 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.020403 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-proxy-tls\") pod \"splitter-graph-7d014-7f8d646646-jnp9w\" (UID: \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\") " pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 13:52:23.105723 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.105531 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 13:52:23.390732 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.390703 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w"] Sep 08 13:52:23.393393 ip-10-0-133-71 kubenswrapper[2554]: W0908 13:52:23.393356 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28e1afb_10ba_4c97_9bfc_83c5bd6f8868.slice/crio-62a522b978f182d140bb9559749997b2ddf8130d7a6b9880dbcd407d372acc8a WatchSource:0}: Error finding container 62a522b978f182d140bb9559749997b2ddf8130d7a6b9880dbcd407d372acc8a: Status 404 returned error can't find the container with id 62a522b978f182d140bb9559749997b2ddf8130d7a6b9880dbcd407d372acc8a Sep 08 13:52:23.636076 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.636037 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" event={"ID":"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868","Type":"ContainerStarted","Data":"20be4d70a1cf9112a41078cc5d024fcd2f90d2349d7a49fdc79f0759c271754e"} Sep 08 13:52:23.636271 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.636091 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" event={"ID":"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868","Type":"ContainerStarted","Data":"62a522b978f182d140bb9559749997b2ddf8130d7a6b9880dbcd407d372acc8a"} Sep 08 13:52:23.636271 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.636182 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 13:52:23.656994 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:23.656884 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" podStartSLOduration=1.6568662010000001 podStartE2EDuration="1.656866201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:52:23.654413467 +0000 UTC m=+2169.987088107" watchObservedRunningTime="2026-09-08 13:52:23.656866201 +0000 UTC m=+2169.989540887" Sep 08 13:52:29.645691 ip-10-0-133-71 kubenswrapper[2554]: I0908 13:52:29.645644 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 14:00:37.448926 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:00:37.448889 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w"] Sep 08 14:00:37.449407 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:00:37.449087 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" containerID="cri-o://20be4d70a1cf9112a41078cc5d024fcd2f90d2349d7a49fdc79f0759c271754e" gracePeriod=30 Sep 08 14:00:39.643547 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:00:39.643504 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:00:44.643903 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:00:44.643857 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:00:49.643440 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:00:49.643401 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:00:49.643865 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:00:49.643525 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 14:00:54.643393 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:00:54.643350 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:00:59.643285 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:00:59.643240 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:01:04.643401 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:04.643363 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:01:07.484204 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:07.484168 2554 generic.go:356] "Generic (PLEG): container finished" podID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerID="20be4d70a1cf9112a41078cc5d024fcd2f90d2349d7a49fdc79f0759c271754e" exitCode=0 Sep 08 14:01:07.484547 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:07.484245 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" event={"ID":"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868","Type":"ContainerDied","Data":"20be4d70a1cf9112a41078cc5d024fcd2f90d2349d7a49fdc79f0759c271754e"} Sep 08 14:01:07.586953 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:07.586928 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 14:01:07.671929 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:07.671893 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-openshift-service-ca-bundle\") pod \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\" (UID: \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\") " Sep 08 14:01:07.672105 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:07.671971 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-proxy-tls\") pod \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\" (UID: \"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868\") " Sep 08 14:01:07.672262 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:07.672237 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" (UID: "d28e1afb-10ba-4c97-9bfc-83c5bd6f8868"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:01:07.673922 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:07.673897 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" (UID: "d28e1afb-10ba-4c97-9bfc-83c5bd6f8868"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:01:07.772480 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:07.772444 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 14:01:07.772480 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:07.772474 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 14:01:08.489291 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:08.489198 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" event={"ID":"d28e1afb-10ba-4c97-9bfc-83c5bd6f8868","Type":"ContainerDied","Data":"62a522b978f182d140bb9559749997b2ddf8130d7a6b9880dbcd407d372acc8a"} Sep 08 14:01:08.489291 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:08.489256 2554 scope.go:117] "RemoveContainer" containerID="20be4d70a1cf9112a41078cc5d024fcd2f90d2349d7a49fdc79f0759c271754e" Sep 08 14:01:08.489820 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:08.489206 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w" Sep 08 14:01:08.505167 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:08.505143 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w"] Sep 08 14:01:08.508749 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:08.508728 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7d014-7f8d646646-jnp9w"] Sep 08 14:01:10.187871 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:01:10.187837 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" path="/var/lib/kubelet/pods/d28e1afb-10ba-4c97-9bfc-83c5bd6f8868/volumes" Sep 08 14:08:07.325611 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:07.325577 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj"] Sep 08 14:08:07.328080 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:07.325803 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" containerID="cri-o://5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6" gracePeriod=30 Sep 08 14:08:10.528983 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:10.528934 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:08:15.529061 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:15.529025 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:08:20.528262 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:20.528226 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:08:20.528636 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:20.528335 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 14:08:24.695916 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:24.695889 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:25.504361 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:25.504328 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:25.528822 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:25.528789 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:08:26.319825 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:26.319794 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:27.113667 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:27.113637 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:27.897563 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:27.897522 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:28.717838 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:28.717806 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:29.554485 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:29.554452 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:30.382822 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:30.382796 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:30.528900 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:30.528862 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:08:31.168370 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:31.168338 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:31.964980 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:31.964951 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:32.746426 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:32.746396 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:33.551012 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:33.550984 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:34.379336 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:34.379305 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:35.223407 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:35.223380 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ce1f6-75dbfb64bd-q4pxj_4297ef51-ddc1-4f75-bc84-1dd4e804854b/switch-graph-ce1f6/0.log" Sep 08 14:08:35.528886 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:35.528843 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:08:37.464989 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:37.464964 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 14:08:37.596742 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:37.596635 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4297ef51-ddc1-4f75-bc84-1dd4e804854b-openshift-service-ca-bundle\") pod \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\" (UID: \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\") " Sep 08 14:08:37.596877 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:37.596858 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4297ef51-ddc1-4f75-bc84-1dd4e804854b-proxy-tls\") pod \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\" (UID: \"4297ef51-ddc1-4f75-bc84-1dd4e804854b\") " Sep 08 14:08:37.597016 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:37.596993 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4297ef51-ddc1-4f75-bc84-1dd4e804854b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4297ef51-ddc1-4f75-bc84-1dd4e804854b" (UID: "4297ef51-ddc1-4f75-bc84-1dd4e804854b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:08:37.598855 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:37.598827 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4297ef51-ddc1-4f75-bc84-1dd4e804854b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4297ef51-ddc1-4f75-bc84-1dd4e804854b" (UID: "4297ef51-ddc1-4f75-bc84-1dd4e804854b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:08:37.698071 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:37.698027 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4297ef51-ddc1-4f75-bc84-1dd4e804854b-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 14:08:37.698071 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:37.698061 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4297ef51-ddc1-4f75-bc84-1dd4e804854b-openshift-service-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\"" Sep 08 14:08:38.035351 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.035316 2554 generic.go:356] "Generic (PLEG): container finished" podID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerID="5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6" exitCode=0 Sep 08 14:08:38.035540 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.035368 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" Sep 08 14:08:38.035540 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.035405 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" event={"ID":"4297ef51-ddc1-4f75-bc84-1dd4e804854b","Type":"ContainerDied","Data":"5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6"} Sep 08 14:08:38.035540 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.035459 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj" event={"ID":"4297ef51-ddc1-4f75-bc84-1dd4e804854b","Type":"ContainerDied","Data":"a276f38b0f1156ae481e0605f726efa3858a1506433d6f242772d19160f35d95"} Sep 08 14:08:38.035540 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.035484 2554 scope.go:117] "RemoveContainer" containerID="5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6" Sep 08 14:08:38.043531 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.043514 2554 scope.go:117] "RemoveContainer" containerID="5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6" Sep 08 14:08:38.043839 ip-10-0-133-71 kubenswrapper[2554]: E0908 14:08:38.043821 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6\": container with ID starting with 5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6 not found: ID does not exist" containerID="5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6" Sep 08 14:08:38.043893 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.043848 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6"} err="failed to get container status \"5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6\": rpc error: code = NotFound desc = could not find container \"5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6\": container with ID starting with 5f1a9d3e8e9bfe61d3ac782f5c44f2f73c5e008cba24c4ddaea473651292beb6 not found: ID does not exist" Sep 08 14:08:38.060868 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.057906 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj"] Sep 08 14:08:38.065487 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.065461 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ce1f6-75dbfb64bd-q4pxj"] Sep 08 14:08:38.187908 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:38.187876 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" path="/var/lib/kubelet/pods/4297ef51-ddc1-4f75-bc84-1dd4e804854b/volumes" Sep 08 14:08:43.308115 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:43.308083 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vl8zn_02ad25a4-76db-4b91-b212-a701afcd24b8/global-pull-secret-syncer/0.log" Sep 08 14:08:43.470941 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:43.470905 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hgc6w_944df588-8dba-4a35-80af-b21714f5ca22/konnectivity-agent/0.log" Sep 08 14:08:43.493165 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:43.493129 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-71.ec2.internal_4d4cca95c4b58a556d25d9ef4b9a311c/haproxy/0.log" Sep 08 14:08:48.363946 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:48.363901 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8dm2_32307348-cbf9-4ccf-a66e-2080f561beb4/node-exporter/0.log" Sep 08 14:08:48.389901 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:48.389875 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8dm2_32307348-cbf9-4ccf-a66e-2080f561beb4/kube-rbac-proxy/0.log" Sep 08 14:08:48.415212 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:48.415186 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8dm2_32307348-cbf9-4ccf-a66e-2080f561beb4/init-textfile/0.log" Sep 08 14:08:51.709399 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.709364 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp"] Sep 08 14:08:51.709802 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.709608 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" Sep 08 14:08:51.709802 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.709619 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" Sep 08 14:08:51.709802 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.709627 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" Sep 08 14:08:51.709802 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.709633 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" Sep 08 14:08:51.709802 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.709693 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d28e1afb-10ba-4c97-9bfc-83c5bd6f8868" containerName="splitter-graph-7d014" Sep 08 14:08:51.709802 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.709704 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4297ef51-ddc1-4f75-bc84-1dd4e804854b" containerName="switch-graph-ce1f6" Sep 08 14:08:51.713887 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.713870 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.716730 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.716701 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p58cr\"/\"openshift-service-ca.crt\"" Sep 08 14:08:51.717759 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.717742 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p58cr\"/\"default-dockercfg-zjqml\"" Sep 08 14:08:51.717887 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.717798 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p58cr\"/\"kube-root-ca.crt\"" Sep 08 14:08:51.722512 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.722490 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp"] Sep 08 14:08:51.805070 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.805037 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-sys\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.805256 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.805086 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-proc\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.805256 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.805155 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp2w\" (UniqueName: \"kubernetes.io/projected/60f684e6-08c8-4141-9585-0056c13938f8-kube-api-access-cvp2w\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.805256 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.805183 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-lib-modules\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.805256 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.805217 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-podres\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.906401 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.906362 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp2w\" (UniqueName: \"kubernetes.io/projected/60f684e6-08c8-4141-9585-0056c13938f8-kube-api-access-cvp2w\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.906401 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.906406 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-lib-modules\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.906640 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.906430 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-podres\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.906640 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.906465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-sys\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.906640 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.906492 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-proc\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.906640 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.906566 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-lib-modules\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.906891 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.906867 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-sys\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.907012 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.906993 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-proc\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.907160 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.907038 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60f684e6-08c8-4141-9585-0056c13938f8-podres\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:51.916918 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:51.916890 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp2w\" (UniqueName: \"kubernetes.io/projected/60f684e6-08c8-4141-9585-0056c13938f8-kube-api-access-cvp2w\") pod \"perf-node-gather-daemonset-m95dp\" (UID: \"60f684e6-08c8-4141-9585-0056c13938f8\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:52.023620 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:52.023582 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:52.295165 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:52.295138 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp"] Sep 08 14:08:52.297303 ip-10-0-133-71 kubenswrapper[2554]: W0908 14:08:52.297276 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod60f684e6_08c8_4141_9585_0056c13938f8.slice/crio-73c11dcbf3388ad6a094d27aec61e4b5d899a5bc53e4c3191b1bb65486a5dcd5 WatchSource:0}: Error finding container 73c11dcbf3388ad6a094d27aec61e4b5d899a5bc53e4c3191b1bb65486a5dcd5: Status 404 returned error can't find the container with id 73c11dcbf3388ad6a094d27aec61e4b5d899a5bc53e4c3191b1bb65486a5dcd5 Sep 08 14:08:53.086428 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:53.086392 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" event={"ID":"60f684e6-08c8-4141-9585-0056c13938f8","Type":"ContainerStarted","Data":"3cca16ae0308f60ce607c0a2421135474c84841fc8ef01343456c236ed80958b"} Sep 08 14:08:53.086826 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:53.086446 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" event={"ID":"60f684e6-08c8-4141-9585-0056c13938f8","Type":"ContainerStarted","Data":"73c11dcbf3388ad6a094d27aec61e4b5d899a5bc53e4c3191b1bb65486a5dcd5"} Sep 08 14:08:53.086826 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:53.086511 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:08:53.107433 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:53.107383 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" podStartSLOduration=2.107368242 podStartE2EDuration="2.107368242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:08:53.105481842 +0000 UTC m=+3159.438156480" watchObservedRunningTime="2026-09-08 14:08:53.107368242 +0000 UTC m=+3159.440042879" Sep 08 14:08:53.231054 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:53.231025 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w9xj9_19ee7d05-e185-443a-ba8e-deb5446dd2ab/dns/0.log" Sep 08 14:08:53.250537 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:53.250505 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w9xj9_19ee7d05-e185-443a-ba8e-deb5446dd2ab/kube-rbac-proxy/0.log" Sep 08 14:08:53.272695 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:53.272648 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hp94q_19f87d80-4c14-4f36-bd27-852618164729/dns-node-resolver/0.log" Sep 08 14:08:53.859990 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:53.859956 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rgj2t_dc121a5f-2eee-4ca2-a204-fcefee21d9ff/node-ca/0.log" Sep 08 14:08:55.049446 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:55.049409 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z69vl_c357275f-3687-4075-bffd-b8acd4f279c4/serve-healthcheck-canary/0.log" Sep 08 14:08:55.627621 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:55.627588 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vvcsd_2cca6b92-400b-4f9e-9c99-0603fd88c093/kube-rbac-proxy/0.log" Sep 08 14:08:55.648085 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:55.648061 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vvcsd_2cca6b92-400b-4f9e-9c99-0603fd88c093/exporter/0.log" Sep 08 14:08:55.670323 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:55.670302 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vvcsd_2cca6b92-400b-4f9e-9c99-0603fd88c093/extractor/0.log" Sep 08 14:08:57.987905 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:57.987874 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6f9bbc7b7c-6wc79_34a620f1-8b55-4418-afb7-931392727606/manager/0.log" Sep 08 14:08:58.009064 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:58.009040 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-bxh55_be21d695-e76e-4e91-aecc-4c4d498e8662/server/0.log" Sep 08 14:08:59.100116 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:08:59.100086 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-m95dp" Sep 08 14:09:04.256744 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.256712 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v6zgx_311ed079-bea5-4b44-b238-fca763ef6788/kube-multus-additional-cni-plugins/0.log" Sep 08 14:09:04.281079 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.281053 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v6zgx_311ed079-bea5-4b44-b238-fca763ef6788/egress-router-binary-copy/0.log" Sep 08 14:09:04.301375 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.301346 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v6zgx_311ed079-bea5-4b44-b238-fca763ef6788/cni-plugins/0.log" Sep 08 14:09:04.325835 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.325804 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v6zgx_311ed079-bea5-4b44-b238-fca763ef6788/bond-cni-plugin/0.log" Sep 08 14:09:04.346762 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.346729 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v6zgx_311ed079-bea5-4b44-b238-fca763ef6788/routeoverride-cni/0.log" Sep 08 14:09:04.373166 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.373132 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v6zgx_311ed079-bea5-4b44-b238-fca763ef6788/whereabouts-cni-bincopy/0.log" Sep 08 14:09:04.392726 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.392698 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v6zgx_311ed079-bea5-4b44-b238-fca763ef6788/whereabouts-cni/0.log" Sep 08 14:09:04.458875 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.458851 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zf6qs_00c3d528-0dd0-4c53-99d3-181afc05cf12/kube-multus/0.log" Sep 08 14:09:04.594418 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.594342 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sghnw_dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3/network-metrics-daemon/0.log" Sep 08 14:09:04.613911 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:04.613882 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sghnw_dbf3cfc3-2d6e-4243-aeed-a87ae931f4f3/kube-rbac-proxy/0.log" Sep 08 14:09:05.811863 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:05.811833 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxmjp_2047f6b0-59c8-4ec1-bfa7-846d78da60b4/ovn-controller/0.log" Sep 08 14:09:05.844788 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:05.844763 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxmjp_2047f6b0-59c8-4ec1-bfa7-846d78da60b4/ovn-acl-logging/0.log" Sep 08 14:09:05.877617 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:05.877587 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxmjp_2047f6b0-59c8-4ec1-bfa7-846d78da60b4/kube-rbac-proxy-node/0.log" Sep 08 14:09:05.903851 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:05.903825 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxmjp_2047f6b0-59c8-4ec1-bfa7-846d78da60b4/kube-rbac-proxy-ovn-metrics/0.log" Sep 08 14:09:05.929515 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:05.929469 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxmjp_2047f6b0-59c8-4ec1-bfa7-846d78da60b4/northd/0.log" Sep 08 14:09:05.957562 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:05.957539 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxmjp_2047f6b0-59c8-4ec1-bfa7-846d78da60b4/nbdb/0.log" Sep 08 14:09:05.981237 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:05.981211 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxmjp_2047f6b0-59c8-4ec1-bfa7-846d78da60b4/sbdb/0.log" Sep 08 14:09:06.033542 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:06.033507 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxmjp_2047f6b0-59c8-4ec1-bfa7-846d78da60b4/ovnkube-controller/0.log" Sep 08 14:09:07.096767 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:07.096731 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-q8rk6_c0eab916-82d1-4c16-8339-12cabd7e0d6c/network-check-target-container/0.log" Sep 08 14:09:08.102009 ip-10-0-133-71 kubenswrapper[2554]: I0908 14:09:08.101983 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kvrtd_98e75a43-fd1d-4b84-b0b6-56cdd64db693/iptables-alerter/0.log"