Sep 08 16:20:45.312244 ip-10-0-128-131 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 08 16:20:45.312253 ip-10-0-128-131 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 08 16:20:45.312260 ip-10-0-128-131 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 08 16:20:45.312420 ip-10-0-128-131 systemd[1]: Failed to start Kubernetes Kubelet. Sep 08 16:20:55.506918 ip-10-0-128-131 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 08 16:20:55.506937 ip-10-0-128-131 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a5fb394ee8684a9a94c800de125ad3b1 -- Sep 08 16:22:52.438628 ip-10-0-128-131 systemd[1]: Starting Kubernetes Kubelet... Sep 08 16:22:52.803517 ip-10-0-128-131 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:22:52.803517 ip-10-0-128-131 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 16:22:52.803517 ip-10-0-128-131 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:22:52.803517 ip-10-0-128-131 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 16:22:52.803517 ip-10-0-128-131 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:22:52.804632 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.804548 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 16:22:52.810457 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810440 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:22:52.810457 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810457 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810461 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810465 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810468 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810471 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810474 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810477 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810480 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810483 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810486 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810488 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810491 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810494 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810496 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810499 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810502 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810504 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810507 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810509 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810512 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:22:52.810528 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810514 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810517 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810519 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810522 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810524 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810533 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810536 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810539 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810541 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810544 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810547 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810550 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810553 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810555 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810558 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810560 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810563 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810565 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810567 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810570 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:22:52.810994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810573 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810575 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810583 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810586 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810588 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810591 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810593 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810596 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810598 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810601 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810603 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810605 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810608 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810610 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810614 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810616 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810619 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810621 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810624 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:22:52.811488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810626 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810629 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810631 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810634 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810636 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810639 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810641 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810645 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810647 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810650 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810652 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810655 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810657 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810659 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810662 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810664 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810667 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810669 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810672 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810674 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:22:52.812079 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810677 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810679 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810681 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810686 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810689 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810692 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810694 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810699 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810702 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810704 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810707 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810709 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810712 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810714 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810716 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810720 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810724 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810726 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.810729 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:22:52.812566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811514 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811521 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811524 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811527 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811530 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811532 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811535 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811538 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811541 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811544 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811547 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811549 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811552 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811555 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811557 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811560 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811562 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811564 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811567 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811569 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:22:52.813043 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811573 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811576 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811578 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811581 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811583 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811586 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811588 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811591 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811594 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811597 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811600 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811603 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811606 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811608 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811611 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811613 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811615 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811618 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811620 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811624 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:22:52.813540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811626 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811629 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811632 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811634 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811637 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811640 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811642 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811645 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811648 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811650 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811653 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811655 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811658 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811660 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811663 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811665 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811668 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811671 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811673 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811676 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:22:52.814036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811678 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811681 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811684 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811686 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811689 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811691 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811694 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811697 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811699 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811701 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811704 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811706 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811709 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811712 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811714 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811717 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811719 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811722 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811725 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811727 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811730 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:22:52.814518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811732 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811737 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811741 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811744 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811747 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811750 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811753 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811755 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811759 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811763 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811768 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811771 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811774 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811777 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811780 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811783 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811785 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.811788 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:22:52.815039 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812182 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812187 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812189 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812193 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812196 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812198 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812201 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812203 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812206 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812209 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812212 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812214 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812217 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812219 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812222 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812224 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812227 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812230 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812233 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812235 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:22:52.815474 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812238 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812240 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812243 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812245 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812248 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812250 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812253 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812255 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812258 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812260 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812263 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812265 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812268 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812270 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812273 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812275 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812278 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812280 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812284 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:22:52.815966 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812286 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812289 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812292 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812296 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812299 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812301 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812304 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812306 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812309 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812312 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812314 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812316 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812319 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812321 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812323 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812326 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812328 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812331 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812333 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812336 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:22:52.816465 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812338 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812341 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812343 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812346 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812349 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812352 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812354 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812357 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812359 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812362 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812364 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812367 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812370 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812372 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812375 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812379 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812382 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812385 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812387 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812390 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:22:52.816981 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812392 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812394 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812397 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812399 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812402 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812406 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812410 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812413 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812416 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812419 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812424 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812427 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812430 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812433 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812435 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812438 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812440 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812443 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812445 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:22:52.817484 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.812448 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812519 2567 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812554 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812560 2567 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812565 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812569 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812572 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812576 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812583 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812587 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812590 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812593 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812597 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812600 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812603 2567 flags.go:64] FLAG: --cgroup-root="" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812606 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812608 2567 flags.go:64] FLAG: --client-ca-file="" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812611 2567 flags.go:64] FLAG: --cloud-provider="external" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812615 2567 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812619 2567 flags.go:64] FLAG: --cluster-domain="" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812622 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812625 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812628 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812631 2567 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 16:22:52.817943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812635 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812638 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812641 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812644 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812647 2567 flags.go:64] FLAG: --contention-profiling="false" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812650 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812652 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812655 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812658 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812662 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812665 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812668 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812671 2567 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812674 2567 flags.go:64] FLAG: --enable-server="true" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812677 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812681 2567 flags.go:64] FLAG: --event-burst="100" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812685 2567 flags.go:64] FLAG: --event-qps="50" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812688 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812691 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812694 2567 flags.go:64] FLAG: --eviction-hard="" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812698 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812700 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812704 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812707 2567 flags.go:64] FLAG: --eviction-soft="" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812710 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 16:22:52.818523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812712 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812717 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812720 2567 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812722 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812725 2567 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812728 2567 flags.go:64] FLAG: --feature-gates="" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812732 2567 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812735 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812738 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812741 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812744 2567 flags.go:64] FLAG: --healthz-port="10248" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812747 2567 flags.go:64] FLAG: --help="false" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812749 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-128-131.ec2.internal" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812752 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812755 2567 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812758 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812761 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812765 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812767 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812770 2567 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812773 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812775 2567 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812778 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812782 2567 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 16:22:52.819148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812785 2567 flags.go:64] FLAG: --kube-reserved="" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812788 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812791 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812795 2567 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812797 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812800 2567 flags.go:64] FLAG: --lock-file="" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812803 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812806 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812809 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812815 2567 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812818 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812820 2567 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812823 2567 flags.go:64] FLAG: --logging-format="text" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812826 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812829 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812832 2567 flags.go:64] FLAG: --manifest-url="" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812834 2567 flags.go:64] FLAG: --manifest-url-header="" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812838 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812841 2567 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812845 2567 flags.go:64] FLAG: --max-pods="110" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812848 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812851 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812854 2567 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812856 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812859 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 16:22:52.819769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812862 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812865 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812871 2567 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812874 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812877 2567 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812880 2567 flags.go:64] FLAG: --pod-cidr="" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812884 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812889 2567 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812891 2567 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812894 2567 flags.go:64] FLAG: --pods-per-core="0" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812898 2567 flags.go:64] FLAG: --port="10250" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812901 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812904 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09cf93a24d429aa04" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812907 2567 flags.go:64] FLAG: --qos-reserved="" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812909 2567 flags.go:64] FLAG: --read-only-port="10255" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812912 2567 flags.go:64] FLAG: --register-node="true" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812917 2567 flags.go:64] FLAG: --register-with-taints="" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812920 2567 flags.go:64] FLAG: --registry-burst="10" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812923 2567 flags.go:64] FLAG: --registry-qps="5" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812926 2567 flags.go:64] FLAG: --reserved-cpus="" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812929 2567 flags.go:64] FLAG: --reserved-memory="" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812932 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812935 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812937 2567 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812940 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 16:22:52.820457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812943 2567 flags.go:64] FLAG: --runonce="false" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812945 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812948 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812951 2567 flags.go:64] FLAG: --seccomp-default="false" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812954 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812957 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812960 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812962 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812965 2567 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812968 2567 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812970 2567 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812973 2567 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812976 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812979 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812983 2567 flags.go:64] FLAG: --system-cgroups="" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812986 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812991 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812994 2567 flags.go:64] FLAG: --tls-cert-file="" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.812997 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813000 2567 flags.go:64] FLAG: --tls-min-version="" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813004 2567 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813007 2567 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813009 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813013 2567 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813029 2567 flags.go:64] FLAG: --v="2" Sep 08 16:22:52.821132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813033 2567 flags.go:64] FLAG: --version="false" Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813042 2567 flags.go:64] FLAG: --vmodule="" Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813046 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813049 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813180 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813184 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813187 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813190 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813193 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813195 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813198 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813201 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813203 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813205 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813208 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813211 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813213 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813216 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813218 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813221 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:22:52.821742 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813223 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813227 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813229 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813232 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813234 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813236 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813239 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813242 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813244 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813247 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813251 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813253 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813257 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813260 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813263 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813265 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813268 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813270 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813273 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813275 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:22:52.822251 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813277 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813280 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813282 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813284 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813287 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813289 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813291 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813294 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813296 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813298 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813302 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813304 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813306 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813309 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813312 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813314 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813317 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813319 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813322 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813326 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:22:52.822738 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813330 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813332 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813337 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813340 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813343 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813345 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813348 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813350 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813353 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813356 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813358 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813361 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813364 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813366 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813369 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813372 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813374 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813377 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813379 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813382 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:22:52.823236 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813384 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813387 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813389 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813392 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813394 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813397 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813399 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813401 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813404 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813406 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813408 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813411 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813414 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813416 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813420 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813422 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813425 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813427 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813429 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:22:52.823737 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813433 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813436 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813439 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.813442 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.813447 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.822199 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.822325 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.822344 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.822350 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822432 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822438 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822441 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822445 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822447 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822450 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:22:52.824247 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822452 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822455 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822458 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822460 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822463 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822465 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822467 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822470 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822473 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822475 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822478 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822480 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822483 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822485 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822488 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822490 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822493 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822495 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822498 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822500 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:22:52.824614 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822503 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822505 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822508 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822517 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822521 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822523 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822526 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822528 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822531 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822535 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822538 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822541 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822544 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822547 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822549 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822552 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822554 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822557 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822559 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:22:52.825114 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822561 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822564 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822567 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822575 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822577 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822580 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822582 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822585 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822587 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822589 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822592 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822594 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822597 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822599 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822602 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822604 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822607 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822616 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822618 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:22:52.825566 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822621 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822625 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822629 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822632 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822635 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822638 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822640 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822643 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822645 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822648 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822650 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822652 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822655 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822657 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822660 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822663 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822665 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822668 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822670 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:22:52.826067 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822673 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822675 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822678 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822680 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822683 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822685 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822687 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822690 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822692 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822694 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822697 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822699 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822707 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822711 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822714 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:22:52.826519 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822716 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.822721 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822865 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822870 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822873 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822876 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822878 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822881 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822884 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822886 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822889 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822891 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822894 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822896 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822899 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:22:52.826889 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822901 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822904 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822906 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822908 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822911 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822913 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822916 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822918 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822921 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822923 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822925 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822928 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822930 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822933 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822935 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822943 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822945 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822949 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822952 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:22:52.827282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822955 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822958 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822961 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822963 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822966 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822969 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822972 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822975 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822977 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822979 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822982 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822984 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822987 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822989 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822992 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822994 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822996 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.822999 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823001 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823004 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:22:52.827736 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823006 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823009 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823011 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823013 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823033 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823037 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823039 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823041 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823044 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823054 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823057 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823059 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823062 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823064 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823066 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823069 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823071 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823074 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823076 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823078 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:22:52.828243 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823081 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823085 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823087 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823090 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823092 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823095 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823097 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823100 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823102 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823104 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823108 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823111 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823114 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823117 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823119 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823122 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823125 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823127 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823130 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:22:52.828723 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823132 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823137 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823140 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823150 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823152 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823155 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823158 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:52.823160 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.823165 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.823279 2567 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.825688 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.826786 2567 server.go:1019] "Starting client certificate rotation" Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.826881 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 16:22:52.829197 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.826912 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 16:22:52.829753 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.829735 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 16:22:52.832274 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.832234 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 16:22:52.850199 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.850180 2567 log.go:25] "Validated CRI v1 runtime API" Sep 08 16:22:52.854304 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.854291 2567 log.go:25] "Validated CRI v1 image API" Sep 08 16:22:52.855143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.855127 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 16:22:52.858225 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.858209 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 16:22:52.858540 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.858521 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 87977f7f-b04c-47f3-a5ff-268c4dc18787:/dev/nvme0n1p3 d3a2bb51-efdd-4901-ae11-3b817a019508:/dev/nvme0n1p4] Sep 08 16:22:52.858576 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.858541 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 16:22:52.863643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.863539 2567 manager.go:216] Machine: {Timestamp:2026-09-08 16:22:52.862113939 +0000 UTC m=+0.333100385 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097688 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20b77bc73ac2b220d54a17f5389751 SystemUUID:ec20b77b-c73a-c2b2-20d5-4a17f5389751 BootID:a5fb394e-e868-4a9a-94c8-00de125ad3b1 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:77:86:cf:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:77:86:cf:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:67:e2:b3:56:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 16:22:52.863643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.863639 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 16:22:52.863756 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.863744 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 16:22:52.867200 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.867176 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 16:22:52.867342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.867203 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-131.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 16:22:52.868071 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.868061 2567 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 16:22:52.868108 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.868074 2567 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 16:22:52.868108 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.868081 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 16:22:52.868904 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.868893 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 16:22:52.870146 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.870136 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 16:22:52.874842 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.874831 2567 state_mem.go:36] "Initialized new in-memory state store" Sep 08 16:22:52.874950 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.874941 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 16:22:52.874983 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.874970 2567 kubelet.go:486] "Attempting to sync node with API server" Sep 08 16:22:52.874983 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.874980 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 16:22:52.875049 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.874993 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 16:22:52.875049 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.875003 2567 kubelet.go:398] "Adding apiserver pod source" Sep 08 16:22:52.875049 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.875012 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 16:22:52.875352 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.875340 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 16:22:52.875400 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.875358 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 16:22:52.877714 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.877693 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 16:22:52.880735 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.880720 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 16:22:52.880794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.880752 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 16:22:52.882113 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882099 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 16:22:52.882162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882120 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 16:22:52.882162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882125 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 16:22:52.882162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882130 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 16:22:52.882162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882139 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 16:22:52.882162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882144 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 16:22:52.882162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882158 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 16:22:52.882162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882163 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 16:22:52.882342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882173 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 16:22:52.882342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882198 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 16:22:52.882342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882226 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 16:22:52.882342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.882240 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 16:22:52.883466 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.883439 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 16:22:52.883742 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.883721 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 16:22:52.883902 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.883790 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kjt9m" Sep 08 16:22:52.885772 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:52.885730 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 16:22:52.885850 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:52.885798 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 16:22:52.885850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.885829 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-131.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 16:22:52.888120 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.888105 2567 server.go:1268] "Started kubelet" Sep 08 16:22:52.888217 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.888185 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 16:22:52.888280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.888188 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 16:22:52.888280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.888263 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 16:22:52.888891 ip-10-0-128-131 systemd[1]: Started Kubernetes Kubelet. Sep 08 16:22:52.889879 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.889861 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 16:22:52.889970 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.889868 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kjt9m" Sep 08 16:22:52.890284 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.890268 2567 server.go:310] "Adding debug handlers to kubelet server" Sep 08 16:22:52.892725 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:52.892696 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 16:22:52.898334 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.898300 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 16:22:52.899036 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.899005 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 16:22:52.899202 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:52.899183 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:52.899861 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.899828 2567 factory.go:55] Registering systemd factory Sep 08 16:22:52.899861 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.899846 2567 factory.go:223] Registration of the systemd container factory successfully Sep 08 16:22:52.900175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.899993 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 16:22:52.900175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.899992 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 16:22:52.900175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.900055 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 16:22:52.900175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.900067 2567 factory.go:153] Registering CRI-O factory Sep 08 16:22:52.900175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.900084 2567 factory.go:223] Registration of the crio container factory successfully Sep 08 16:22:52.900175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.900140 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 16:22:52.900175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.900171 2567 factory.go:103] Registering Raw factory Sep 08 16:22:52.900175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.900178 2567 reconstruct.go:97] "Volume reconstruction finished" Sep 08 16:22:52.900715 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.900186 2567 manager.go:1196] Started watching for new ooms in manager Sep 08 16:22:52.900715 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.900186 2567 reconciler.go:29] "Reconciler: start to sync state" Sep 08 16:22:52.900715 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.900658 2567 manager.go:349] Starting recovery of all containers Sep 08 16:22:52.902221 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.902199 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:22:52.904940 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:52.904913 2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-131.ec2.internal\" not found" node="ip-10-0-128-131.ec2.internal" Sep 08 16:22:52.912107 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.912082 2567 manager.go:354] Recovery completed Sep 08 16:22:52.917029 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.917000 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:22:52.918592 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.918577 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:22:52.918649 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.918606 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:22:52.918649 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.918619 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:22:52.919143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.919130 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 16:22:52.919143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.919139 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 16:22:52.919250 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.919148 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:22:52.919250 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.919157 2567 state_mem.go:36] "Initialized new in-memory state store" Sep 08 16:22:52.919788 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.919772 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:22:52.919835 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.919802 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:22:52.919835 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.919812 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:22:52.921409 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.921396 2567 policy_none.go:49] "None policy: Start" Sep 08 16:22:52.921474 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.921414 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 16:22:52.921474 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.921429 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 16:22:52.923628 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.923612 2567 policy_none.go:47] "Start" Sep 08 16:22:52.959147 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.959133 2567 manager.go:337] "Starting Device Plugin manager" Sep 08 16:22:52.959217 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:52.959180 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 16:22:52.959217 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.959194 2567 server.go:87] "Starting device plugin registration server" Sep 08 16:22:52.959451 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.959435 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 16:22:52.959532 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.959459 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 16:22:52.959641 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.959561 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 16:22:52.959641 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.959636 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 16:22:52.959733 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:52.959645 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 16:22:52.962114 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:52.962097 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 16:22:52.962199 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:52.962139 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.044678 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.044642 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 16:22:53.045996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.045978 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 16:22:53.046072 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.046038 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 16:22:53.046072 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.046062 2567 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 16:22:53.046154 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.046100 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 16:22:53.048079 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.048062 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:22:53.060249 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.060186 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:22:53.060995 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.060980 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:22:53.061062 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.061009 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:22:53.061062 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.061038 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:22:53.061125 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.061077 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.068765 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.068753 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.068819 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.068772 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-131.ec2.internal\": node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.086245 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.086222 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.146961 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.146921 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"] Sep 08 16:22:53.147069 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.147038 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:22:53.148463 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.148444 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:22:53.148558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.148472 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:22:53.148558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.148482 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:22:53.150721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.150709 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:22:53.150853 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.150837 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.150890 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.150867 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:22:53.151436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.151421 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:22:53.151516 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.151444 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:22:53.151516 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.151456 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:22:53.151516 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.151487 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:22:53.151607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.151517 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:22:53.151607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.151531 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:22:53.153619 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.153605 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.153661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.153631 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:22:53.154299 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.154276 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:22:53.154379 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.154304 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:22:53.154379 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.154313 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:22:53.174541 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.174525 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-131.ec2.internal\" not found" node="ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.178900 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.178887 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-131.ec2.internal\" not found" node="ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.186945 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.186930 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.288005 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.287975 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.301269 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.301249 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f25ceed021acd67419e766168870646-config\") pod \"kube-apiserver-proxy-ip-10-0-128-131.ec2.internal\" (UID: \"8f25ceed021acd67419e766168870646\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.301324 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.301274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17d1cd933af78237c2c7c1c9296435f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"17d1cd933af78237c2c7c1c9296435f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.301324 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.301293 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17d1cd933af78237c2c7c1c9296435f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"17d1cd933af78237c2c7c1c9296435f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.388454 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.388397 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.401758 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.401729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17d1cd933af78237c2c7c1c9296435f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"17d1cd933af78237c2c7c1c9296435f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.401822 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.401761 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f25ceed021acd67419e766168870646-config\") pod \"kube-apiserver-proxy-ip-10-0-128-131.ec2.internal\" (UID: \"8f25ceed021acd67419e766168870646\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.401822 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.401780 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17d1cd933af78237c2c7c1c9296435f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"17d1cd933af78237c2c7c1c9296435f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.401822 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.401806 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17d1cd933af78237c2c7c1c9296435f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"17d1cd933af78237c2c7c1c9296435f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.401960 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.401839 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f25ceed021acd67419e766168870646-config\") pod \"kube-apiserver-proxy-ip-10-0-128-131.ec2.internal\" (UID: \"8f25ceed021acd67419e766168870646\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.401960 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.401897 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17d1cd933af78237c2c7c1c9296435f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"17d1cd933af78237c2c7c1c9296435f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.476913 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.476862 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.481926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.481772 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.489400 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.489383 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.590034 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.589995 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.690571 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.690487 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.750934 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.750908 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:22:53.790901 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.790865 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found" Sep 08 16:22:53.820981 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.820955 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:22:53.826866 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.826851 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 16:22:53.826994 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.826975 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:22:53.827087 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.827011 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:22:53.827087 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.827014 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:22:53.827087 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.827041 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:22:53.876171 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.876138 2567 apiserver.go:52] "Watching apiserver" Sep 08 16:22:53.893768 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.893748 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 16:22:53.894698 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.894675 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cqvbk","kube-system/konnectivity-agent-td9m4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg","openshift-cluster-node-tuning-operator/tuned-vtsjm","openshift-dns/node-resolver-dkg7m","openshift-image-registry/node-ca-5grg2","openshift-multus/multus-additional-cni-plugins-h589z","openshift-multus/network-metrics-daemon-426b4","openshift-multus/multus-4rh8c","openshift-network-diagnostics/network-check-target-hp42x","openshift-network-operator/iptables-alerter-tr84q","openshift-ovn-kubernetes/ovnkube-node-4vfmw"] Sep 08 16:22:53.896221 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.896184 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 16:17:53 +0000 UTC" deadline="2028-03-03 02:06:28.756565067 +0000 UTC" Sep 08 16:22:53.896221 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.896219 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12993h43m34.86034781s" Sep 08 16:22:53.897173 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.897156 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:53.897270 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.897251 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:22:53.899183 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.899124 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 16:22:53.899280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.899215 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:22:53.899419 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.899398 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.901224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.901207 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 16:22:53.901342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.901298 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c7w2t\"" Sep 08 16:22:53.901427 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.901413 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:53.901494 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.901476 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 16:22:53.903483 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.903466 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:53.903566 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.903544 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 16:22:53.903677 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.903661 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 16:22:53.903766 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.903747 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c5cvj\"" Sep 08 16:22:53.903851 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.903837 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 16:22:53.904311 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904295 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-registration-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:53.904366 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904318 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-device-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:53.904366 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904335 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:53.904366 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904363 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9ac825f-6d94-455d-a06b-2f3674222f6c-agent-certs\") pod \"konnectivity-agent-td9m4\" (UID: \"f9ac825f-6d94-455d-a06b-2f3674222f6c\") " pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:22:53.904459 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904393 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-socket-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:53.904459 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904421 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bdzd\" (UniqueName: \"kubernetes.io/projected/9626347a-952a-4e28-9a20-7661d89cc440-kube-api-access-2bdzd\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:53.904459 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-kubelet-config\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:53.904552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904484 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-dbus\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:53.904552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904506 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:53.904552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904539 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-etc-selinux\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:53.904643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904561 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-sys-fs\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:53.904643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.904614 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9ac825f-6d94-455d-a06b-2f3674222f6c-konnectivity-ca\") pod \"konnectivity-agent-td9m4\" (UID: \"f9ac825f-6d94-455d-a06b-2f3674222f6c\") " pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:22:53.905583 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.905564 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:53.905691 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.905676 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jqfln\"" Sep 08 16:22:53.905755 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.905712 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:22:53.905755 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.905734 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 16:22:53.907646 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.907628 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 16:22:53.907719 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.907648 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sprr8\"" Sep 08 16:22:53.907719 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.907679 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 16:22:53.907868 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.907853 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:53.908234 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.908217 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 16:22:53.910075 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.910054 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 16:22:53.910160 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.910097 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 16:22:53.910160 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.910147 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 16:22:53.910261 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.910166 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 16:22:53.910261 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.910112 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:53.910559 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.910545 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m9lbc\"" Sep 08 16:22:53.910722 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.910709 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" Sep 08 16:22:53.912331 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.912312 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 16:22:53.912420 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.912365 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 16:22:53.912420 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.912370 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 16:22:53.912522 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.912444 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:53.912577 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.912536 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:22:53.912630 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.912586 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9mhs7\"" Sep 08 16:22:53.912684 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.912648 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 16:22:53.912684 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.912658 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 16:22:53.914701 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.914683 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4rh8c" Sep 08 16:22:53.916588 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.916569 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zqmnp\"" Sep 08 16:22:53.916898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.916876 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 16:22:53.917178 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.917161 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:22:53.917266 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:53.917238 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:22:53.919557 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.919540 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 16:22:53.919719 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.919704 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:53.921818 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.921798 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 16:22:53.921912 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.921834 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 16:22:53.921912 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.921839 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vkmxp\"" Sep 08 16:22:53.921912 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.921834 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:22:53.922085 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.921954 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"] Sep 08 16:22:53.922085 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.921981 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"] Sep 08 16:22:53.922085 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.922061 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:53.923912 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.923893 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bwr7c\"" Sep 08 16:22:53.924143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.924128 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 16:22:53.924206 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.924193 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 16:22:53.924850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.924830 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 16:22:53.924939 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.924883 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 16:22:53.924939 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.924899 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 16:22:53.924939 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.924932 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 16:22:53.936504 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.936484 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jf82v" Sep 08 16:22:53.945048 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:53.945013 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jf82v" Sep 08 16:22:54.001469 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.001447 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 16:22:54.005757 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005735 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-lib-modules\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.005846 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005766 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41f534ff-2f6d-47bc-adcc-792705333dc2-system-cni-dir\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.005846 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005790 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41f534ff-2f6d-47bc-adcc-792705333dc2-cnibin\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.005846 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005826 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:54.005962 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005858 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-run-netns\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.005962 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-run-k8s-cni-cncf-io\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.005962 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005907 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-socket-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.005962 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005930 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bdzd\" (UniqueName: \"kubernetes.io/projected/9626347a-952a-4e28-9a20-7661d89cc440-kube-api-access-2bdzd\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.006127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005968 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-run\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.006127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.005988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40daeca6-4624-45bf-831f-51e77aab369c-hosts-file\") pod \"node-resolver-dkg7m\" (UID: \"40daeca6-4624-45bf-831f-51e77aab369c\") " pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.006127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17ca8c4a-6567-4610-9166-f37f64cb40a5-ovnkube-script-lib\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.006127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006071 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-cni-dir\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.006127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006095 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fb2e9d1-28c2-489d-b85f-d096cfbad814-cni-binary-copy\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.006127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006113 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-daemon-config\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.006317 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdfpw\" (UniqueName: \"kubernetes.io/projected/40daeca6-4624-45bf-831f-51e77aab369c-kube-api-access-bdfpw\") pod \"node-resolver-dkg7m\" (UID: \"40daeca6-4624-45bf-831f-51e77aab369c\") " pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.006317 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006184 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz27n\" (UniqueName: \"kubernetes.io/projected/41f534ff-2f6d-47bc-adcc-792705333dc2-kube-api-access-bz27n\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.006317 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006204 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp2bc\" (UniqueName: \"kubernetes.io/projected/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-kube-api-access-cp2bc\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:54.006317 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq4rf\" (UniqueName: \"kubernetes.io/projected/17ca8c4a-6567-4610-9166-f37f64cb40a5-kube-api-access-wq4rf\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.006317 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006251 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-system-cni-dir\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.006317 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4fq\" (UniqueName: \"kubernetes.io/projected/5fb2e9d1-28c2-489d-b85f-d096cfbad814-kube-api-access-kh4fq\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.006317 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9ac825f-6d94-455d-a06b-2f3674222f6c-konnectivity-ca\") pod \"konnectivity-agent-td9m4\" (UID: \"f9ac825f-6d94-455d-a06b-2f3674222f6c\") " pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006322 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-systemd\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006337 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-systemd-units\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006351 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-var-lib-openvswitch\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006383 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-sysconfig\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006405 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e987a7b-3158-45a6-b031-a4dedaf8c20b-tmp\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006427 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ec20b4a-239d-49aa-b713-e65f0ba60775-host\") pod \"node-ca-5grg2\" (UID: \"3ec20b4a-239d-49aa-b713-e65f0ba60775\") " pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006452 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-socket-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006445 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ec20b4a-239d-49aa-b713-e65f0ba60775-serviceca\") pod \"node-ca-5grg2\" (UID: \"3ec20b4a-239d-49aa-b713-e65f0ba60775\") " pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006509 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcjp\" (UniqueName: \"kubernetes.io/projected/3ec20b4a-239d-49aa-b713-e65f0ba60775-kube-api-access-vvcjp\") pod \"node-ca-5grg2\" (UID: \"3ec20b4a-239d-49aa-b713-e65f0ba60775\") " pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006536 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-conf-dir\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-kubelet-config\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006592 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41f534ff-2f6d-47bc-adcc-792705333dc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006620 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.006640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006637 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-kubelet-config\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006643 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-cni-netd\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006680 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-cnibin\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-hostroot\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006715 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-etc-selinux\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006739 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-modprobe-d\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006754 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-log-socket\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006770 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-device-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006776 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9ac825f-6d94-455d-a06b-2f3674222f6c-konnectivity-ca\") pod \"konnectivity-agent-td9m4\" (UID: \"f9ac825f-6d94-455d-a06b-2f3674222f6c\") " pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2pz\" (UniqueName: \"kubernetes.io/projected/1e987a7b-3158-45a6-b031-a4dedaf8c20b-kube-api-access-gf2pz\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006811 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41f534ff-2f6d-47bc-adcc-792705333dc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006841 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-run-ovn\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.007170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.006830 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-etc-selinux\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-device-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007414 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfc4\" (UniqueName: \"kubernetes.io/projected/e0f2ccc0-369f-4dd0-9614-685858dd61b3-kube-api-access-mdfc4\") pod \"iptables-alerter-tr84q\" (UID: \"e0f2ccc0-369f-4dd0-9614-685858dd61b3\") " pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007481 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-sys\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007524 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41f534ff-2f6d-47bc-adcc-792705333dc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007553 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17ca8c4a-6567-4610-9166-f37f64cb40a5-ovn-node-metrics-cert\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007584 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0f2ccc0-369f-4dd0-9614-685858dd61b3-iptables-alerter-script\") pod \"iptables-alerter-tr84q\" (UID: \"e0f2ccc0-369f-4dd0-9614-685858dd61b3\") " pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007608 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0f2ccc0-369f-4dd0-9614-685858dd61b3-host-slash\") pod \"iptables-alerter-tr84q\" (UID: \"e0f2ccc0-369f-4dd0-9614-685858dd61b3\") " pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-dbus\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:54.007680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-var-lib-kubelet\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-run-systemd\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-sysctl-d\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-node-log\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007805 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17ca8c4a-6567-4610-9166-f37f64cb40a5-ovnkube-config\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007833 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-var-lib-kubelet\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007862 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-run-multus-certs\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007886 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-etc-kubernetes\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007917 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-registration-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007946 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.007974 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-host\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.008004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41f534ff-2f6d-47bc-adcc-792705333dc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.008049 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-kubelet\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.008078 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-slash\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.008224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.008105 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-cni-bin\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.008863 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.008288 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-dbus\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:54.008863 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.008362 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.008863 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.008423 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-registration-dir\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.008863 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.008593 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:22:54.008863 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.008685 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret podName:a3e8e45e-b772-4cc4-87c8-f9222bdd5488 nodeName:}" failed. No retries permitted until 2026-09-08 16:22:54.508657229 +0000 UTC m=+1.979643677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret") pod "global-pull-secret-syncer-cqvbk" (UID: "a3e8e45e-b772-4cc4-87c8-f9222bdd5488") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:22:54.009124 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.008998 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:22:54.009124 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9ac825f-6d94-455d-a06b-2f3674222f6c-agent-certs\") pod \"konnectivity-agent-td9m4\" (UID: \"f9ac825f-6d94-455d-a06b-2f3674222f6c\") " pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:22:54.009124 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009091 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-tuned\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.009124 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009120 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-var-lib-cni-bin\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.009300 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009148 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-var-lib-cni-multus\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.009300 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009176 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-kubernetes\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.009300 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009202 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-sysctl-conf\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.009300 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40daeca6-4624-45bf-831f-51e77aab369c-tmp-dir\") pod \"node-resolver-dkg7m\" (UID: \"40daeca6-4624-45bf-831f-51e77aab369c\") " pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.009300 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009251 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-run-openvswitch\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.009300 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009279 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17ca8c4a-6567-4610-9166-f37f64cb40a5-env-overrides\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.009547 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-sys-fs\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.009547 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-etc-openvswitch\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.009547 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009358 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-socket-dir-parent\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.009547 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009385 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-run-netns\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.009720 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.009673 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9626347a-952a-4e28-9a20-7661d89cc440-sys-fs\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.010243 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.010222 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 16:22:54.013845 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.013712 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9ac825f-6d94-455d-a06b-2f3674222f6c-agent-certs\") pod \"konnectivity-agent-td9m4\" (UID: \"f9ac825f-6d94-455d-a06b-2f3674222f6c\") " pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:22:54.014091 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.014075 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bdzd\" (UniqueName: \"kubernetes.io/projected/9626347a-952a-4e28-9a20-7661d89cc440-kube-api-access-2bdzd\") pod \"aws-ebs-csi-driver-node-rbcfg\" (UID: \"9626347a-952a-4e28-9a20-7661d89cc440\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.109635 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109607 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-etc-openvswitch\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.109635 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-socket-dir-parent\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109652 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-run-netns\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109668 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-lib-modules\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109740 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-etc-openvswitch\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-lib-modules\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109762 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-run-netns\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109773 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-socket-dir-parent\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41f534ff-2f6d-47bc-adcc-792705333dc2-system-cni-dir\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109822 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41f534ff-2f6d-47bc-adcc-792705333dc2-cnibin\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109778 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41f534ff-2f6d-47bc-adcc-792705333dc2-system-cni-dir\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.109905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41f534ff-2f6d-47bc-adcc-792705333dc2-cnibin\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.109963 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109969 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-run-netns\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.109995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-run-k8s-cni-cncf-io\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.110050 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs podName:bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29 nodeName:}" failed. No retries permitted until 2026-09-08 16:22:54.610009908 +0000 UTC m=+2.080996357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs") pod "network-metrics-daemon-426b4" (UID: "bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110055 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-run-k8s-cni-cncf-io\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110067 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-run\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110096 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40daeca6-4624-45bf-831f-51e77aab369c-hosts-file\") pod \"node-resolver-dkg7m\" (UID: \"40daeca6-4624-45bf-831f-51e77aab369c\") " pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110051 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-run-netns\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110111 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-run\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110122 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17ca8c4a-6567-4610-9166-f37f64cb40a5-ovnkube-script-lib\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-cni-dir\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fb2e9d1-28c2-489d-b85f-d096cfbad814-cni-binary-copy\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-daemon-config\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110208 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfpw\" (UniqueName: \"kubernetes.io/projected/40daeca6-4624-45bf-831f-51e77aab369c-kube-api-access-bdfpw\") pod \"node-resolver-dkg7m\" (UID: \"40daeca6-4624-45bf-831f-51e77aab369c\") " pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40daeca6-4624-45bf-831f-51e77aab369c-hosts-file\") pod \"node-resolver-dkg7m\" (UID: \"40daeca6-4624-45bf-831f-51e77aab369c\") " pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.110358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110216 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-cni-dir\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz27n\" (UniqueName: \"kubernetes.io/projected/41f534ff-2f6d-47bc-adcc-792705333dc2-kube-api-access-bz27n\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp2bc\" (UniqueName: \"kubernetes.io/projected/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-kube-api-access-cp2bc\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110358 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq4rf\" (UniqueName: \"kubernetes.io/projected/17ca8c4a-6567-4610-9166-f37f64cb40a5-kube-api-access-wq4rf\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110381 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-system-cni-dir\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4fq\" (UniqueName: \"kubernetes.io/projected/5fb2e9d1-28c2-489d-b85f-d096cfbad814-kube-api-access-kh4fq\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-systemd\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110454 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-systemd-units\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110480 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-var-lib-openvswitch\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-sysconfig\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110551 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-var-lib-openvswitch\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110559 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-systemd\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110596 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-systemd-units\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110635 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-system-cni-dir\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110641 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-sysconfig\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110709 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fb2e9d1-28c2-489d-b85f-d096cfbad814-cni-binary-copy\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110740 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e987a7b-3158-45a6-b031-a4dedaf8c20b-tmp\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110748 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-daemon-config\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110756 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17ca8c4a-6567-4610-9166-f37f64cb40a5-ovnkube-script-lib\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110787 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ec20b4a-239d-49aa-b713-e65f0ba60775-host\") pod \"node-ca-5grg2\" (UID: \"3ec20b4a-239d-49aa-b713-e65f0ba60775\") " pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110830 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ec20b4a-239d-49aa-b713-e65f0ba60775-serviceca\") pod \"node-ca-5grg2\" (UID: \"3ec20b4a-239d-49aa-b713-e65f0ba60775\") " pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcjp\" (UniqueName: \"kubernetes.io/projected/3ec20b4a-239d-49aa-b713-e65f0ba60775-kube-api-access-vvcjp\") pod \"node-ca-5grg2\" (UID: \"3ec20b4a-239d-49aa-b713-e65f0ba60775\") " pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110834 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ec20b4a-239d-49aa-b713-e65f0ba60775-host\") pod \"node-ca-5grg2\" (UID: \"3ec20b4a-239d-49aa-b713-e65f0ba60775\") " pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110891 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-conf-dir\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110931 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-multus-conf-dir\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110944 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41f534ff-2f6d-47bc-adcc-792705333dc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.110977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111003 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-cni-netd\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111047 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-cnibin\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111066 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111089 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-cni-netd\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111110 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-cnibin\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111160 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-hostroot\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-modprobe-d\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-log-socket\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111241 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2pz\" (UniqueName: \"kubernetes.io/projected/1e987a7b-3158-45a6-b031-a4dedaf8c20b-kube-api-access-gf2pz\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.111903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111257 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-hostroot\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111295 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-log-socket\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111298 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41f534ff-2f6d-47bc-adcc-792705333dc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111333 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-modprobe-d\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111335 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ec20b4a-239d-49aa-b713-e65f0ba60775-serviceca\") pod \"node-ca-5grg2\" (UID: \"3ec20b4a-239d-49aa-b713-e65f0ba60775\") " pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111395 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-run-ovn\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111422 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfc4\" (UniqueName: \"kubernetes.io/projected/e0f2ccc0-369f-4dd0-9614-685858dd61b3-kube-api-access-mdfc4\") pod \"iptables-alerter-tr84q\" (UID: \"e0f2ccc0-369f-4dd0-9614-685858dd61b3\") " pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-sys\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111451 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-run-ovn\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111488 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41f534ff-2f6d-47bc-adcc-792705333dc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111560 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17ca8c4a-6567-4610-9166-f37f64cb40a5-ovn-node-metrics-cert\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111554 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41f534ff-2f6d-47bc-adcc-792705333dc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111586 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-sys\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111665 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41f534ff-2f6d-47bc-adcc-792705333dc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0f2ccc0-369f-4dd0-9614-685858dd61b3-iptables-alerter-script\") pod \"iptables-alerter-tr84q\" (UID: \"e0f2ccc0-369f-4dd0-9614-685858dd61b3\") " pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.112781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111726 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41f534ff-2f6d-47bc-adcc-792705333dc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111737 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0f2ccc0-369f-4dd0-9614-685858dd61b3-host-slash\") pod \"iptables-alerter-tr84q\" (UID: \"e0f2ccc0-369f-4dd0-9614-685858dd61b3\") " pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111763 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-var-lib-kubelet\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111799 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0f2ccc0-369f-4dd0-9614-685858dd61b3-host-slash\") pod \"iptables-alerter-tr84q\" (UID: \"e0f2ccc0-369f-4dd0-9614-685858dd61b3\") " pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-run-systemd\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111841 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-sysctl-d\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111814 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-var-lib-kubelet\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111846 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-run-systemd\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111863 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-node-log\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111897 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-node-log\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17ca8c4a-6567-4610-9166-f37f64cb40a5-ovnkube-config\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111943 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-var-lib-kubelet\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.111986 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-run-multus-certs\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112012 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-etc-kubernetes\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112050 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-var-lib-kubelet\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112087 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-host\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112120 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41f534ff-2f6d-47bc-adcc-792705333dc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-run-multus-certs\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.113436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112148 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-kubelet\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112185 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-slash\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112200 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-kubelet\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-etc-kubernetes\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112274 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0f2ccc0-369f-4dd0-9614-685858dd61b3-iptables-alerter-script\") pod \"iptables-alerter-tr84q\" (UID: \"e0f2ccc0-369f-4dd0-9614-685858dd61b3\") " pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112312 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-host\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112085 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-sysctl-d\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112399 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-cni-bin\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112418 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17ca8c4a-6567-4610-9166-f37f64cb40a5-ovnkube-config\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112480 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-cni-bin\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112486 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-host-slash\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112508 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-tuned\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-var-lib-cni-bin\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-var-lib-cni-multus\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112592 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-kubernetes\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112613 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-sysctl-conf\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40daeca6-4624-45bf-831f-51e77aab369c-tmp-dir\") pod \"node-resolver-dkg7m\" (UID: \"40daeca6-4624-45bf-831f-51e77aab369c\") " pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.113986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112656 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-run-openvswitch\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112680 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17ca8c4a-6567-4610-9166-f37f64cb40a5-env-overrides\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112701 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-kubernetes\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-var-lib-cni-bin\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112783 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41f534ff-2f6d-47bc-adcc-792705333dc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112786 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fb2e9d1-28c2-489d-b85f-d096cfbad814-host-var-lib-cni-multus\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-sysctl-conf\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.112975 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca8c4a-6567-4610-9166-f37f64cb40a5-run-openvswitch\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.113086 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40daeca6-4624-45bf-831f-51e77aab369c-tmp-dir\") pod \"node-resolver-dkg7m\" (UID: \"40daeca6-4624-45bf-831f-51e77aab369c\") " pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.113259 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17ca8c4a-6567-4610-9166-f37f64cb40a5-env-overrides\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.113585 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e987a7b-3158-45a6-b031-a4dedaf8c20b-tmp\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.114523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.114123 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17ca8c4a-6567-4610-9166-f37f64cb40a5-ovn-node-metrics-cert\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.114847 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.114534 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e987a7b-3158-45a6-b031-a4dedaf8c20b-etc-tuned\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.118932 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.118335 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:22:54.118932 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.118361 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:22:54.118932 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.118375 2567 projected.go:196] Error preparing data for projected volume kube-api-access-2sq8s for pod openshift-network-diagnostics/network-check-target-hp42x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:22:54.118932 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.118441 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s podName:a4e974fd-6a6f-40b1-a3a4-c213ad566b23 nodeName:}" failed. No retries permitted until 2026-09-08 16:22:54.618424259 +0000 UTC m=+2.089410713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2sq8s" (UniqueName: "kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s") pod "network-check-target-hp42x" (UID: "a4e974fd-6a6f-40b1-a3a4-c213ad566b23") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:22:54.119549 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.119524 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfpw\" (UniqueName: \"kubernetes.io/projected/40daeca6-4624-45bf-831f-51e77aab369c-kube-api-access-bdfpw\") pod \"node-resolver-dkg7m\" (UID: \"40daeca6-4624-45bf-831f-51e77aab369c\") " pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.120257 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.120231 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2pz\" (UniqueName: \"kubernetes.io/projected/1e987a7b-3158-45a6-b031-a4dedaf8c20b-kube-api-access-gf2pz\") pod \"tuned-vtsjm\" (UID: \"1e987a7b-3158-45a6-b031-a4dedaf8c20b\") " pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.120517 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.120500 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4fq\" (UniqueName: \"kubernetes.io/projected/5fb2e9d1-28c2-489d-b85f-d096cfbad814-kube-api-access-kh4fq\") pod \"multus-4rh8c\" (UID: \"5fb2e9d1-28c2-489d-b85f-d096cfbad814\") " pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.120650 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.120631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp2bc\" (UniqueName: \"kubernetes.io/projected/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-kube-api-access-cp2bc\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:54.120769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.120753 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcjp\" (UniqueName: \"kubernetes.io/projected/3ec20b4a-239d-49aa-b713-e65f0ba60775-kube-api-access-vvcjp\") pod \"node-ca-5grg2\" (UID: \"3ec20b4a-239d-49aa-b713-e65f0ba60775\") " pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.121006 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.120989 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz27n\" (UniqueName: \"kubernetes.io/projected/41f534ff-2f6d-47bc-adcc-792705333dc2-kube-api-access-bz27n\") pod \"multus-additional-cni-plugins-h589z\" (UID: \"41f534ff-2f6d-47bc-adcc-792705333dc2\") " pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.121089 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.121013 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq4rf\" (UniqueName: \"kubernetes.io/projected/17ca8c4a-6567-4610-9166-f37f64cb40a5-kube-api-access-wq4rf\") pod \"ovnkube-node-4vfmw\" (UID: \"17ca8c4a-6567-4610-9166-f37f64cb40a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.121405 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.121387 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfc4\" (UniqueName: \"kubernetes.io/projected/e0f2ccc0-369f-4dd0-9614-685858dd61b3-kube-api-access-mdfc4\") pod \"iptables-alerter-tr84q\" (UID: \"e0f2ccc0-369f-4dd0-9614-685858dd61b3\") " pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.169062 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.168991 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d1cd933af78237c2c7c1c9296435f2.slice/crio-5f98b34f7b5cff16ce99096efa9c3dd9b6b117b1cdd350137e920c08fa6d2679 WatchSource:0}: Error finding container 5f98b34f7b5cff16ce99096efa9c3dd9b6b117b1cdd350137e920c08fa6d2679: Status 404 returned error can't find the container with id 5f98b34f7b5cff16ce99096efa9c3dd9b6b117b1cdd350137e920c08fa6d2679 Sep 08 16:22:54.173614 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.173595 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:22:54.225847 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.225760 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:22:54.235718 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.235686 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ac825f_6d94_455d_a06b_2f3674222f6c.slice/crio-2ff67e8a864c026244b7f3ae3d95c2ec521d2423584b0751a3f3c9d5cf0860ed WatchSource:0}: Error finding container 2ff67e8a864c026244b7f3ae3d95c2ec521d2423584b0751a3f3c9d5cf0860ed: Status 404 returned error can't find the container with id 2ff67e8a864c026244b7f3ae3d95c2ec521d2423584b0751a3f3c9d5cf0860ed Sep 08 16:22:54.241031 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.240998 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" Sep 08 16:22:54.249132 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.249108 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9626347a_952a_4e28_9a20_7661d89cc440.slice/crio-9c31fefefc41158b9faa5b1f36ee1084c119a7fd96705301e09119df7e582eda WatchSource:0}: Error finding container 9c31fefefc41158b9faa5b1f36ee1084c119a7fd96705301e09119df7e582eda: Status 404 returned error can't find the container with id 9c31fefefc41158b9faa5b1f36ee1084c119a7fd96705301e09119df7e582eda Sep 08 16:22:54.261363 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.261343 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" Sep 08 16:22:54.264129 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.264110 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dkg7m" Sep 08 16:22:54.270253 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.270233 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e987a7b_3158_45a6_b031_a4dedaf8c20b.slice/crio-21f9c1f383519499ffdfb74416ae46db1a5bd62d9d5d84026ed652a8a4694c7a WatchSource:0}: Error finding container 21f9c1f383519499ffdfb74416ae46db1a5bd62d9d5d84026ed652a8a4694c7a: Status 404 returned error can't find the container with id 21f9c1f383519499ffdfb74416ae46db1a5bd62d9d5d84026ed652a8a4694c7a Sep 08 16:22:54.272282 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.272263 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40daeca6_4624_45bf_831f_51e77aab369c.slice/crio-ed968440d371b162caf95c8c61439a8bcef2b8a864a547f0092b34e9f3ddf8a1 WatchSource:0}: Error finding container ed968440d371b162caf95c8c61439a8bcef2b8a864a547f0092b34e9f3ddf8a1: Status 404 returned error can't find the container with id ed968440d371b162caf95c8c61439a8bcef2b8a864a547f0092b34e9f3ddf8a1 Sep 08 16:22:54.290656 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.290629 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5grg2" Sep 08 16:22:54.296678 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.296653 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h589z" Sep 08 16:22:54.302536 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.302509 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec20b4a_239d_49aa_b713_e65f0ba60775.slice/crio-64c1e22f1cfdc98d127c3a05cdb903ebd9c0786b4136cd1bad595d730a02e13f WatchSource:0}: Error finding container 64c1e22f1cfdc98d127c3a05cdb903ebd9c0786b4136cd1bad595d730a02e13f: Status 404 returned error can't find the container with id 64c1e22f1cfdc98d127c3a05cdb903ebd9c0786b4136cd1bad595d730a02e13f Sep 08 16:22:54.305246 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.305223 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f534ff_2f6d_47bc_adcc_792705333dc2.slice/crio-02e33a42a3012e83a898baa881bd6acee591187bf3ab0c5c2e872c2f3b443566 WatchSource:0}: Error finding container 02e33a42a3012e83a898baa881bd6acee591187bf3ab0c5c2e872c2f3b443566: Status 404 returned error can't find the container with id 02e33a42a3012e83a898baa881bd6acee591187bf3ab0c5c2e872c2f3b443566 Sep 08 16:22:54.312067 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.312047 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4rh8c" Sep 08 16:22:54.321374 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.321352 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb2e9d1_28c2_489d_b85f_d096cfbad814.slice/crio-296cd4ae52d38cceadae0849b0679a269e12b4c0a5cc8689e29d5c7186aa6c68 WatchSource:0}: Error finding container 296cd4ae52d38cceadae0849b0679a269e12b4c0a5cc8689e29d5c7186aa6c68: Status 404 returned error can't find the container with id 296cd4ae52d38cceadae0849b0679a269e12b4c0a5cc8689e29d5c7186aa6c68 Sep 08 16:22:54.322855 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.322840 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tr84q" Sep 08 16:22:54.326156 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.326132 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:22:54.332799 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.332778 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f2ccc0_369f_4dd0_9614_685858dd61b3.slice/crio-fc1225807cf6d2637c2340819673c1d704d98c772f138a28656b1c67bec8e688 WatchSource:0}: Error finding container fc1225807cf6d2637c2340819673c1d704d98c772f138a28656b1c67bec8e688: Status 404 returned error can't find the container with id fc1225807cf6d2637c2340819673c1d704d98c772f138a28656b1c67bec8e688 Sep 08 16:22:54.335959 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.335932 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ca8c4a_6567_4610_9166_f37f64cb40a5.slice/crio-e7b09b4a8729a048461c111c95d1405fee794ed842403d46baba8d3c7ad2fe2b WatchSource:0}: Error finding container e7b09b4a8729a048461c111c95d1405fee794ed842403d46baba8d3c7ad2fe2b: Status 404 returned error can't find the container with id e7b09b4a8729a048461c111c95d1405fee794ed842403d46baba8d3c7ad2fe2b Sep 08 16:22:54.516163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.516077 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:54.516285 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.516199 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:22:54.516285 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.516250 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret podName:a3e8e45e-b772-4cc4-87c8-f9222bdd5488 nodeName:}" failed. No retries permitted until 2026-09-08 16:22:55.51623804 +0000 UTC m=+2.987224474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret") pod "global-pull-secret-syncer-cqvbk" (UID: "a3e8e45e-b772-4cc4-87c8-f9222bdd5488") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:22:54.558994 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:22:54.558970 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f25ceed021acd67419e766168870646.slice/crio-7d7ad2c818af9f7bad6e33d4750728867962201d5098862df5a42b53cfbbd57d WatchSource:0}: Error finding container 7d7ad2c818af9f7bad6e33d4750728867962201d5098862df5a42b53cfbbd57d: Status 404 returned error can't find the container with id 7d7ad2c818af9f7bad6e33d4750728867962201d5098862df5a42b53cfbbd57d Sep 08 16:22:54.616948 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.616918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:54.617129 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.617078 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:22:54.617195 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.617135 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs podName:bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29 nodeName:}" failed. No retries permitted until 2026-09-08 16:22:55.617121874 +0000 UTC m=+3.088108322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs") pod "network-metrics-daemon-426b4" (UID: "bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:22:54.717738 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.717700 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:22:54.717921 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.717875 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:22:54.717921 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.717912 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:22:54.718011 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.717926 2567 projected.go:196] Error preparing data for projected volume kube-api-access-2sq8s for pod openshift-network-diagnostics/network-check-target-hp42x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:22:54.718011 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:54.717997 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s podName:a4e974fd-6a6f-40b1-a3a4-c213ad566b23 nodeName:}" failed. No retries permitted until 2026-09-08 16:22:55.717976382 +0000 UTC m=+3.188962841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sq8s" (UniqueName: "kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s") pod "network-check-target-hp42x" (UID: "a4e974fd-6a6f-40b1-a3a4-c213ad566b23") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:22:54.808245 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.808165 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:22:54.948445 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.947741 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:17:54 +0000 UTC" deadline="2028-04-07 00:24:05.510946078 +0000 UTC" Sep 08 16:22:54.948445 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:54.947808 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13832h1m10.563141529s" Sep 08 16:22:55.063215 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.063072 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" event={"ID":"8f25ceed021acd67419e766168870646","Type":"ContainerStarted","Data":"7d7ad2c818af9f7bad6e33d4750728867962201d5098862df5a42b53cfbbd57d"} Sep 08 16:22:55.069623 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.069523 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tr84q" event={"ID":"e0f2ccc0-369f-4dd0-9614-685858dd61b3","Type":"ContainerStarted","Data":"fc1225807cf6d2637c2340819673c1d704d98c772f138a28656b1c67bec8e688"} Sep 08 16:22:55.074468 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.074384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4rh8c" event={"ID":"5fb2e9d1-28c2-489d-b85f-d096cfbad814","Type":"ContainerStarted","Data":"296cd4ae52d38cceadae0849b0679a269e12b4c0a5cc8689e29d5c7186aa6c68"} Sep 08 16:22:55.079989 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.079905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" event={"ID":"1e987a7b-3158-45a6-b031-a4dedaf8c20b","Type":"ContainerStarted","Data":"21f9c1f383519499ffdfb74416ae46db1a5bd62d9d5d84026ed652a8a4694c7a"} Sep 08 16:22:55.082792 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.082691 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" event={"ID":"9626347a-952a-4e28-9a20-7661d89cc440","Type":"ContainerStarted","Data":"9c31fefefc41158b9faa5b1f36ee1084c119a7fd96705301e09119df7e582eda"} Sep 08 16:22:55.099283 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.099198 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-td9m4" event={"ID":"f9ac825f-6d94-455d-a06b-2f3674222f6c","Type":"ContainerStarted","Data":"2ff67e8a864c026244b7f3ae3d95c2ec521d2423584b0751a3f3c9d5cf0860ed"} Sep 08 16:22:55.120054 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.118167 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"17d1cd933af78237c2c7c1c9296435f2","Type":"ContainerStarted","Data":"5f98b34f7b5cff16ce99096efa9c3dd9b6b117b1cdd350137e920c08fa6d2679"} Sep 08 16:22:55.130112 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.130062 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerStarted","Data":"e7b09b4a8729a048461c111c95d1405fee794ed842403d46baba8d3c7ad2fe2b"} Sep 08 16:22:55.137829 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.137804 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h589z" event={"ID":"41f534ff-2f6d-47bc-adcc-792705333dc2","Type":"ContainerStarted","Data":"02e33a42a3012e83a898baa881bd6acee591187bf3ab0c5c2e872c2f3b443566"} Sep 08 16:22:55.139546 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.139520 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5grg2" event={"ID":"3ec20b4a-239d-49aa-b713-e65f0ba60775","Type":"ContainerStarted","Data":"64c1e22f1cfdc98d127c3a05cdb903ebd9c0786b4136cd1bad595d730a02e13f"} Sep 08 16:22:55.147654 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.147589 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dkg7m" event={"ID":"40daeca6-4624-45bf-831f-51e77aab369c","Type":"ContainerStarted","Data":"ed968440d371b162caf95c8c61439a8bcef2b8a864a547f0092b34e9f3ddf8a1"} Sep 08 16:22:55.224991 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.224758 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:22:55.525807 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.525706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:55.525987 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:55.525931 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:22:55.526063 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:55.526007 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret podName:a3e8e45e-b772-4cc4-87c8-f9222bdd5488 nodeName:}" failed. No retries permitted until 2026-09-08 16:22:57.525985563 +0000 UTC m=+4.996972024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret") pod "global-pull-secret-syncer-cqvbk" (UID: "a3e8e45e-b772-4cc4-87c8-f9222bdd5488") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:22:55.626834 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.626797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:55.627039 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:55.626967 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:22:55.627138 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:55.627042 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs podName:bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29 nodeName:}" failed. No retries permitted until 2026-09-08 16:22:57.627023037 +0000 UTC m=+5.098009490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs") pod "network-metrics-daemon-426b4" (UID: "bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:22:55.728075 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.727968 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:22:55.728282 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:55.728172 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:22:55.728282 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:55.728194 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:22:55.728282 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:55.728207 2567 projected.go:196] Error preparing data for projected volume kube-api-access-2sq8s for pod openshift-network-diagnostics/network-check-target-hp42x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:22:55.728282 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:55.728268 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s podName:a4e974fd-6a6f-40b1-a3a4-c213ad566b23 nodeName:}" failed. No retries permitted until 2026-09-08 16:22:57.728248931 +0000 UTC m=+5.199235382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sq8s" (UniqueName: "kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s") pod "network-check-target-hp42x" (UID: "a4e974fd-6a6f-40b1-a3a4-c213ad566b23") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:22:55.948562 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.948452 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:17:54 +0000 UTC" deadline="2028-02-09 10:41:10.770200746 +0000 UTC" Sep 08 16:22:55.948562 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:55.948497 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12450h18m14.821709202s" Sep 08 16:22:56.046726 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:56.046685 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:22:56.046935 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:56.046685 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:56.046935 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:56.046841 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:56.046935 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:56.046836 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:22:56.047176 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:56.046973 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:22:56.047176 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:56.047101 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:22:56.402827 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:56.402724 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:22:56.763161 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:56.762771 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:22:57.549250 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:57.549208 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:57.549920 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:57.549409 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:22:57.549920 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:57.549476 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret podName:a3e8e45e-b772-4cc4-87c8-f9222bdd5488 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:01.549457824 +0000 UTC m=+9.020444266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret") pod "global-pull-secret-syncer-cqvbk" (UID: "a3e8e45e-b772-4cc4-87c8-f9222bdd5488") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:22:57.650674 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:57.650627 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:57.650909 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:57.650887 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:22:57.650978 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:57.650969 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs podName:bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:01.650945521 +0000 UTC m=+9.121931965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs") pod "network-metrics-daemon-426b4" (UID: "bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:22:57.751499 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:57.751456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:22:57.751743 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:57.751699 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:22:57.751743 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:57.751738 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:22:57.751871 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:57.751755 2567 projected.go:196] Error preparing data for projected volume kube-api-access-2sq8s for pod openshift-network-diagnostics/network-check-target-hp42x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:22:57.751871 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:57.751823 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s podName:a4e974fd-6a6f-40b1-a3a4-c213ad566b23 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:01.751801829 +0000 UTC m=+9.222788273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sq8s" (UniqueName: "kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s") pod "network-check-target-hp42x" (UID: "a4e974fd-6a6f-40b1-a3a4-c213ad566b23") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:22:58.046401 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:58.046310 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:22:58.046548 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:58.046456 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:22:58.046907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:58.046886 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:22:58.047021 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:58.047000 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:22:58.047062 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:22:58.047055 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:22:58.047199 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:22:58.047142 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:00.046782 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:00.046741 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:00.047322 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:00.046878 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:00.047322 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:00.047162 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:00.047322 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:00.047170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:00.047322 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:00.047299 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:00.047557 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:00.047387 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:01.585446 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:01.585404 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:01.585944 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:01.585641 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:23:01.585944 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:01.585709 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret podName:a3e8e45e-b772-4cc4-87c8-f9222bdd5488 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:09.585689757 +0000 UTC m=+17.056676193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret") pod "global-pull-secret-syncer-cqvbk" (UID: "a3e8e45e-b772-4cc4-87c8-f9222bdd5488") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:23:01.686187 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:01.686091 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:01.686389 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:01.686265 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:23:01.686389 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:01.686354 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs podName:bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:09.686332937 +0000 UTC m=+17.157319388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs") pod "network-metrics-daemon-426b4" (UID: "bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:23:01.786797 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:01.786726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:01.786998 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:01.786898 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:23:01.786998 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:01.786924 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:23:01.786998 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:01.786938 2567 projected.go:196] Error preparing data for projected volume kube-api-access-2sq8s for pod openshift-network-diagnostics/network-check-target-hp42x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:23:01.787206 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:01.787000 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s podName:a4e974fd-6a6f-40b1-a3a4-c213ad566b23 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:09.786981514 +0000 UTC m=+17.257967954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sq8s" (UniqueName: "kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s") pod "network-check-target-hp42x" (UID: "a4e974fd-6a6f-40b1-a3a4-c213ad566b23") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:23:02.046522 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:02.046432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:02.046522 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:02.046478 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:02.046755 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:02.046576 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:02.046755 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:02.046592 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:02.046755 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:02.046663 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:02.046918 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:02.046784 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:04.046581 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:04.046533 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:04.047131 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:04.046657 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:04.047131 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:04.047055 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:04.047261 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:04.047155 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:04.047261 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:04.047171 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:04.047362 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:04.047285 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:06.047058 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:06.047005 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:06.047058 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:06.047044 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:06.047572 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:06.047005 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:06.047572 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:06.047145 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:06.047572 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:06.047219 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:06.047572 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:06.047308 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:08.046638 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:08.046609 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:08.047182 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:08.046603 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:08.047182 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:08.046728 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:08.047182 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:08.046811 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:08.047182 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:08.046611 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:08.047182 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:08.046914 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:09.648184 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:09.648141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:09.648609 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:09.648257 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:23:09.648609 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:09.648329 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret podName:a3e8e45e-b772-4cc4-87c8-f9222bdd5488 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:25.648315188 +0000 UTC m=+33.119301640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret") pod "global-pull-secret-syncer-cqvbk" (UID: "a3e8e45e-b772-4cc4-87c8-f9222bdd5488") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:23:09.748974 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:09.748934 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:09.749175 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:09.749141 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:23:09.749243 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:09.749225 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs podName:bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:25.749202377 +0000 UTC m=+33.220188831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs") pod "network-metrics-daemon-426b4" (UID: "bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:23:09.849394 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:09.849356 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:09.849559 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:09.849510 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:23:09.849559 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:09.849530 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:23:09.849559 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:09.849539 2567 projected.go:196] Error preparing data for projected volume kube-api-access-2sq8s for pod openshift-network-diagnostics/network-check-target-hp42x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:23:09.849697 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:09.849593 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s podName:a4e974fd-6a6f-40b1-a3a4-c213ad566b23 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:25.849578309 +0000 UTC m=+33.320564742 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sq8s" (UniqueName: "kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s") pod "network-check-target-hp42x" (UID: "a4e974fd-6a6f-40b1-a3a4-c213ad566b23") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:23:10.046676 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:10.046632 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:10.046676 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:10.046683 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:10.046874 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:10.046681 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:10.046874 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:10.046770 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:10.046951 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:10.046893 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:10.047039 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:10.046965 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:12.046362 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:12.046320 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:12.046815 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:12.046320 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:12.046815 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:12.046460 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:12.046815 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:12.046320 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:12.046815 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:12.046521 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:12.046815 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:12.046617 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:14.047410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:14.046909 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:14.047410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:14.046948 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:14.047410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:14.046960 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:14.047410 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:14.047070 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:14.047410 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:14.047168 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:14.047410 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:14.047250 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:15.198785 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:15.198605 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" event={"ID":"1e987a7b-3158-45a6-b031-a4dedaf8c20b","Type":"ContainerStarted","Data":"57e42536e901e2e77233f74c7da2133320c650314bba47365ca0390dfeb8859a"} Sep 08 16:23:15.200240 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:15.200216 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerStarted","Data":"c286f18755dab41f7810e52fa522ed483cfe3587e0e18f1f5e2c963955da8d1d"} Sep 08 16:23:15.202598 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:15.202510 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" event={"ID":"8f25ceed021acd67419e766168870646","Type":"ContainerStarted","Data":"ae27a719240c49e3c6e0d9fd27d6f60a5e111f11c42b5b585a3b972bc37ab60b"} Sep 08 16:23:15.228618 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:15.228248 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vtsjm" podStartSLOduration=1.6403857579999999 podStartE2EDuration="22.228228836s" totalImagesPullingTime="20.587843078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:15.227536665 +0000 UTC m=+22.698523142" watchObservedRunningTime="2026-09-08 16:23:15.228228836 +0000 UTC m=+22.699215292" Sep 08 16:23:15.239860 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:15.239813 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" podStartSLOduration=21.239793436 podStartE2EDuration="21.239793436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:15.239416326 +0000 UTC m=+22.710402784" watchObservedRunningTime="2026-09-08 16:23:15.239793436 +0000 UTC m=+22.710779893" Sep 08 16:23:16.046724 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.046428 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:16.046724 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.046457 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:16.046927 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:16.046719 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:16.046927 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.046461 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:16.046927 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:16.046823 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:16.047121 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:16.046938 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:16.205840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.205805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4rh8c" event={"ID":"5fb2e9d1-28c2-489d-b85f-d096cfbad814","Type":"ContainerStarted","Data":"a699fd14e3376d5432986c675066af57bf82faf11eba8b35e4debbb02ec745fd"} Sep 08 16:23:16.207105 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.207085 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" event={"ID":"9626347a-952a-4e28-9a20-7661d89cc440","Type":"ContainerStarted","Data":"0048026537a4b11d79b4f5be03471d9d75be1ab484c5972f74be2977134360d1"} Sep 08 16:23:16.208497 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.208471 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-td9m4" event={"ID":"f9ac825f-6d94-455d-a06b-2f3674222f6c","Type":"ContainerStarted","Data":"013ade10c720795118d6b5d46bf8a90cd39a5a7aee3ff68292dc02fb0907a045"} Sep 08 16:23:16.209871 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.209849 2567 generic.go:356] "Generic (PLEG): container finished" podID="17d1cd933af78237c2c7c1c9296435f2" containerID="f449bf3771b618bb5135354c87652e7a37429bb9a051d2fd3b64a64d1927092a" exitCode=0 Sep 08 16:23:16.209952 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.209929 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"17d1cd933af78237c2c7c1c9296435f2","Type":"ContainerDied","Data":"f449bf3771b618bb5135354c87652e7a37429bb9a051d2fd3b64a64d1927092a"} Sep 08 16:23:16.214595 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.214577 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:23:16.214889 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.214871 2567 generic.go:356] "Generic (PLEG): container finished" podID="17ca8c4a-6567-4610-9166-f37f64cb40a5" containerID="c501aab2c74a1f80556e1b6f1e8524dc5a0fa9f812ed7d698ad8b274dd2acc49" exitCode=1 Sep 08 16:23:16.214964 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.214937 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerStarted","Data":"69226099aae1b8a0c5f746630aa8f3a858662a0bc0442c0393800c4a7f609ebb"} Sep 08 16:23:16.215042 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.215010 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerStarted","Data":"2d9a485eccaa7c7ae948b33a7d0cdf39c1c627a1718c9ffdc07bc3213031af8a"} Sep 08 16:23:16.215097 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.215084 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerStarted","Data":"7e9537ee04793c4b43df370661a565bfc6fba84aadf6795f683d8a3d8b2be1f9"} Sep 08 16:23:16.215150 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.215137 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerStarted","Data":"cc22dcc2d808e3466785ae825e4da7431c0e4edf1a55ef051e59e8ce3e0ed537"} Sep 08 16:23:16.215195 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.215174 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerDied","Data":"c501aab2c74a1f80556e1b6f1e8524dc5a0fa9f812ed7d698ad8b274dd2acc49"} Sep 08 16:23:16.216254 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.216236 2567 generic.go:356] "Generic (PLEG): container finished" podID="41f534ff-2f6d-47bc-adcc-792705333dc2" containerID="e3f1248f686f0941d9107e7447951f30879b09885a49cea258798e81c0b522cd" exitCode=0 Sep 08 16:23:16.216350 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.216304 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h589z" event={"ID":"41f534ff-2f6d-47bc-adcc-792705333dc2","Type":"ContainerDied","Data":"e3f1248f686f0941d9107e7447951f30879b09885a49cea258798e81c0b522cd"} Sep 08 16:23:16.217589 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.217566 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5grg2" event={"ID":"3ec20b4a-239d-49aa-b713-e65f0ba60775","Type":"ContainerStarted","Data":"6a7ad7777d97e6dc4aac2ad04e187c26d60538f644683e67129f0ceafab4278d"} Sep 08 16:23:16.218904 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.218882 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dkg7m" event={"ID":"40daeca6-4624-45bf-831f-51e77aab369c","Type":"ContainerStarted","Data":"736d5b44a59a2e6fbb16811a19f9053db122b6ce6fc7d0885f3edf24d9ba42f4"} Sep 08 16:23:16.235628 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.235588 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5grg2" podStartSLOduration=2.680819664 podStartE2EDuration="23.235576292s" totalImagesPullingTime="20.554756628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:16.235386825 +0000 UTC m=+23.706373281" watchObservedRunningTime="2026-09-08 16:23:16.235576292 +0000 UTC m=+23.706562747" Sep 08 16:23:16.235788 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.235770 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4rh8c" podStartSLOduration=2.366507177 podStartE2EDuration="23.235761918s" totalImagesPullingTime="20.869254741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:16.22342821 +0000 UTC m=+23.694414666" watchObservedRunningTime="2026-09-08 16:23:16.235761918 +0000 UTC m=+23.706748374" Sep 08 16:23:16.270692 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.270647 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-td9m4" podStartSLOduration=2.649502176 podStartE2EDuration="23.270632409s" totalImagesPullingTime="20.621130233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:16.270284969 +0000 UTC m=+23.741271426" watchObservedRunningTime="2026-09-08 16:23:16.270632409 +0000 UTC m=+23.741618864" Sep 08 16:23:16.285142 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.285076 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dkg7m" podStartSLOduration=2.700576776 podStartE2EDuration="23.285054612s" totalImagesPullingTime="20.584477836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:16.284580648 +0000 UTC m=+23.755567116" watchObservedRunningTime="2026-09-08 16:23:16.285054612 +0000 UTC m=+23.756041131" Sep 08 16:23:16.603597 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.603440 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 16:23:16.808382 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.808352 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:23:16.808882 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.808869 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:23:16.972890 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.972796 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T16:23:16.603587068Z","UUID":"b441abf9-761c-45f2-abde-e09d5b3b3683","Handler":null,"Name":"","Endpoint":""} Sep 08 16:23:16.974280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.974263 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 16:23:16.974375 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:16.974290 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 16:23:17.223684 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:17.223600 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" event={"ID":"9626347a-952a-4e28-9a20-7661d89cc440","Type":"ContainerStarted","Data":"32e64b5dbff6b0c10427efdbd0770d714b20da08619f535c8dcaad2873e4fe0a"} Sep 08 16:23:17.225779 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:17.225743 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"17d1cd933af78237c2c7c1c9296435f2","Type":"ContainerStarted","Data":"1d26bd69e9592398f0255e102302f2f845a33b13b1398495e096a1749aeb8f9c"} Sep 08 16:23:17.227369 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:17.227341 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tr84q" event={"ID":"e0f2ccc0-369f-4dd0-9614-685858dd61b3","Type":"ContainerStarted","Data":"5bb61b5bb6fef51069168eac71462843c20827e4e74befd345e002f1d033c3e2"} Sep 08 16:23:17.227648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:17.227626 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:23:17.228150 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:17.228131 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-td9m4" Sep 08 16:23:17.241584 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:17.241531 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" podStartSLOduration=23.241511314 podStartE2EDuration="23.241511314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:17.24064372 +0000 UTC m=+24.711630175" watchObservedRunningTime="2026-09-08 16:23:17.241511314 +0000 UTC m=+24.712497770" Sep 08 16:23:17.272538 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:17.272481 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tr84q" podStartSLOduration=3.747883481 podStartE2EDuration="24.272459544s" totalImagesPullingTime="20.524576063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:17.271892353 +0000 UTC m=+24.742878811" watchObservedRunningTime="2026-09-08 16:23:17.272459544 +0000 UTC m=+24.743446001" Sep 08 16:23:18.047008 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:18.046927 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:18.047008 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:18.046956 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:18.047222 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:18.046927 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:18.047222 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:18.047077 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:18.047222 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:18.047166 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:18.047316 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:18.047261 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:18.231385 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:18.231352 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" event={"ID":"9626347a-952a-4e28-9a20-7661d89cc440","Type":"ContainerStarted","Data":"848a14c482bd811922e50e760d6e0eb4d6944e020436bd829cb6f842679f957d"} Sep 08 16:23:18.234480 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:18.234460 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:23:18.234915 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:18.234890 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerStarted","Data":"69e94fdc70e7c81982e36ac7af1e323a7a750efac88f417b101f846c83de14b1"} Sep 08 16:23:18.248096 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:18.248038 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbcfg" podStartSLOduration=2.777620327 podStartE2EDuration="25.248000771s" totalImagesPullingTime="22.470380444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:18.247796206 +0000 UTC m=+25.718782662" watchObservedRunningTime="2026-09-08 16:23:18.248000771 +0000 UTC m=+25.718987229" Sep 08 16:23:19.517963 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:19.517932 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dkg7m_40daeca6-4624-45bf-831f-51e77aab369c/dns-node-resolver/0.log" Sep 08 16:23:20.047273 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:20.047236 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:20.047273 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:20.047282 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:20.047516 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:20.047358 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:20.047516 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:20.047362 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:20.047516 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:20.047502 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:20.047643 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:20.047583 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:20.243579 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:20.243550 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:23:20.244061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:20.244005 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerStarted","Data":"a2151975493619bf29c43094b1a534279d4f37cb1dc0ec0b072f013b04ed4076"} Sep 08 16:23:20.244354 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:20.244315 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:23:20.244566 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:20.244548 2567 scope.go:117] "RemoveContainer" containerID="c501aab2c74a1f80556e1b6f1e8524dc5a0fa9f812ed7d698ad8b274dd2acc49" Sep 08 16:23:20.263039 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:20.262866 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:23:20.503234 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:20.503155 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5grg2_3ec20b4a-239d-49aa-b713-e65f0ba60775/node-ca/0.log" Sep 08 16:23:21.251764 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:21.251741 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:23:21.252462 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:21.252206 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" event={"ID":"17ca8c4a-6567-4610-9166-f37f64cb40a5","Type":"ContainerStarted","Data":"560bc65e562962ecf8d68bc46a4cc0744191aa528d3b52158e75a7d1e7cfc43b"} Sep 08 16:23:21.252694 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:21.252673 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:23:21.252778 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:21.252767 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:23:21.270314 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:21.270283 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:23:21.286476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:21.286406 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" podStartSLOduration=7.760706843 podStartE2EDuration="28.286381465s" totalImagesPullingTime="20.525674622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:21.281347721 +0000 UTC m=+28.752334180" watchObservedRunningTime="2026-09-08 16:23:21.286381465 +0000 UTC m=+28.757367921" Sep 08 16:23:22.046580 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:22.046495 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:22.046733 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:22.046512 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:22.046733 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:22.046607 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:22.046733 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:22.046613 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:22.046733 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:22.046682 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:22.046958 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:22.046753 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:23.258840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:23.258805 2567 generic.go:356] "Generic (PLEG): container finished" podID="41f534ff-2f6d-47bc-adcc-792705333dc2" containerID="015164fb7cd9deed7e2e31053777351e0382ab7bda5e707ab0ea51010dce29d2" exitCode=0 Sep 08 16:23:23.259486 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:23.258899 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h589z" event={"ID":"41f534ff-2f6d-47bc-adcc-792705333dc2","Type":"ContainerDied","Data":"015164fb7cd9deed7e2e31053777351e0382ab7bda5e707ab0ea51010dce29d2"} Sep 08 16:23:24.046281 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:24.046257 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:24.046383 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:24.046362 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:24.046437 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:24.046365 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:24.046437 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:24.046424 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:24.046520 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:24.046500 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:24.046581 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:24.046567 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:24.263262 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:24.263182 2567 generic.go:356] "Generic (PLEG): container finished" podID="41f534ff-2f6d-47bc-adcc-792705333dc2" containerID="384d40ea46f0cbc31caf6cd572efbf3b4dd2ed3a95dbab3c94d2b8687db99bba" exitCode=0 Sep 08 16:23:24.263262 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:24.263232 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h589z" event={"ID":"41f534ff-2f6d-47bc-adcc-792705333dc2","Type":"ContainerDied","Data":"384d40ea46f0cbc31caf6cd572efbf3b4dd2ed3a95dbab3c94d2b8687db99bba"} Sep 08 16:23:25.267541 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:25.267457 2567 generic.go:356] "Generic (PLEG): container finished" podID="41f534ff-2f6d-47bc-adcc-792705333dc2" containerID="dda7a52ae81c627c4a11e2060006b9a7a26ed8499d4030ca13c7de742943ae2f" exitCode=0 Sep 08 16:23:25.267541 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:25.267520 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h589z" event={"ID":"41f534ff-2f6d-47bc-adcc-792705333dc2","Type":"ContainerDied","Data":"dda7a52ae81c627c4a11e2060006b9a7a26ed8499d4030ca13c7de742943ae2f"} Sep 08 16:23:25.665774 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:25.665751 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:25.665906 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:25.665890 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:23:25.665959 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:25.665952 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret podName:a3e8e45e-b772-4cc4-87c8-f9222bdd5488 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:57.665935909 +0000 UTC m=+65.136922360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret") pod "global-pull-secret-syncer-cqvbk" (UID: "a3e8e45e-b772-4cc4-87c8-f9222bdd5488") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:23:25.766808 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:25.766766 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:25.766988 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:25.766920 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:23:25.767071 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:25.766999 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs podName:bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:57.76698324 +0000 UTC m=+65.237969690 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs") pod "network-metrics-daemon-426b4" (UID: "bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:23:25.867800 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:25.867764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:25.867979 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:25.867906 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:23:25.867979 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:25.867931 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:23:25.867979 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:25.867945 2567 projected.go:196] Error preparing data for projected volume kube-api-access-2sq8s for pod openshift-network-diagnostics/network-check-target-hp42x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:23:25.868147 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:25.868006 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s podName:a4e974fd-6a6f-40b1-a3a4-c213ad566b23 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:57.867985088 +0000 UTC m=+65.338971522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sq8s" (UniqueName: "kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s") pod "network-check-target-hp42x" (UID: "a4e974fd-6a6f-40b1-a3a4-c213ad566b23") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:23:26.046373 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:26.046297 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:26.046373 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:26.046339 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:26.046586 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:26.046343 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:26.046586 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:26.046415 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:26.046586 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:26.046522 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:26.046706 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:26.046610 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:28.046552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:28.046507 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:28.046552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:28.046538 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:28.047099 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:28.046538 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:28.047099 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:28.046639 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:28.047099 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:28.046725 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:28.047099 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:28.046761 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:30.047265 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:30.047225 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:30.047265 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:30.047270 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:30.047652 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:30.047275 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:30.047652 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:30.047347 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:30.047652 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:30.047468 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:30.047652 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:30.047533 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:31.284120 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:31.284083 2567 generic.go:356] "Generic (PLEG): container finished" podID="41f534ff-2f6d-47bc-adcc-792705333dc2" containerID="ff4ff379c925127f25a3a9d13f156f2eb2b1d889f5a6e1480529503921d08498" exitCode=0 Sep 08 16:23:31.284545 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:31.284126 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h589z" event={"ID":"41f534ff-2f6d-47bc-adcc-792705333dc2","Type":"ContainerDied","Data":"ff4ff379c925127f25a3a9d13f156f2eb2b1d889f5a6e1480529503921d08498"} Sep 08 16:23:32.046703 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:32.046678 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:32.046859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:32.046680 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:32.046977 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:32.046678 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:32.049530 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:32.047078 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:32.049530 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:32.047165 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:32.049530 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:32.047258 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:32.290144 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:32.290104 2567 generic.go:356] "Generic (PLEG): container finished" podID="41f534ff-2f6d-47bc-adcc-792705333dc2" containerID="9ca9a0062a4a18ea674b9937f8ab195a126beb6bdb1f217819253f2ebf0d1663" exitCode=0 Sep 08 16:23:32.290490 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:32.290160 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h589z" event={"ID":"41f534ff-2f6d-47bc-adcc-792705333dc2","Type":"ContainerDied","Data":"9ca9a0062a4a18ea674b9937f8ab195a126beb6bdb1f217819253f2ebf0d1663"} Sep 08 16:23:33.296171 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:33.295952 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h589z" event={"ID":"41f534ff-2f6d-47bc-adcc-792705333dc2","Type":"ContainerStarted","Data":"2f10e11526240757d670a54a8f5d06e4c59cd97102626e7d1dc4078c2d737ee0"} Sep 08 16:23:33.316782 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:33.316731 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h589z" podStartSLOduration=6.98780321 podStartE2EDuration="40.31671632s" totalImagesPullingTime="33.32891311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:33.315250555 +0000 UTC m=+40.786237010" watchObservedRunningTime="2026-09-08 16:23:33.31671632 +0000 UTC m=+40.787702775" Sep 08 16:23:34.046932 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:34.046890 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:34.047144 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:34.046901 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:34.047144 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:34.047003 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:34.047144 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:34.047119 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:34.047144 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:34.046901 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:34.047295 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:34.047220 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:36.046617 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:36.046581 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:36.046617 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:36.046610 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:36.047005 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:36.046682 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:36.047005 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:36.046722 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:36.047005 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:36.046789 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:36.047005 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:36.046871 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:38.047117 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:38.047073 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:38.047630 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:38.047089 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:38.047630 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:38.047089 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:38.047630 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:38.047302 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:38.047630 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:38.047346 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:38.047630 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:38.047174 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:40.046486 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:40.046456 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:40.046875 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:40.046490 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:40.046875 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:40.046459 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:40.046875 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:40.046558 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:40.046875 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:40.046635 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:40.046875 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:40.046690 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:42.047065 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:42.047033 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:42.047065 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:42.047052 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:42.047453 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:42.047045 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:42.047453 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:42.047147 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:42.047453 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:42.047190 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:42.047453 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:42.047298 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:42.817661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:42.817628 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cqvbk"] Sep 08 16:23:42.817828 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:42.817785 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:42.817930 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:42.817895 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:42.821154 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:42.820797 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hp42x"] Sep 08 16:23:42.821154 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:42.820913 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:42.821154 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:42.820994 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:42.821523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:42.821502 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-426b4"] Sep 08 16:23:42.821601 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:42.821590 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:42.821684 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:42.821668 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:44.046918 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:44.046881 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:44.047323 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:44.046882 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:44.047323 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:44.047038 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:44.047323 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:44.047085 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:45.046525 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:45.046496 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:45.046700 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:45.046638 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-426b4" podUID="bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29" Sep 08 16:23:46.046893 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.046856 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:46.047288 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.046856 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:46.047288 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:46.046958 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hp42x" podUID="a4e974fd-6a6f-40b1-a3a4-c213ad566b23" Sep 08 16:23:46.047288 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:46.047053 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqvbk" podUID="a3e8e45e-b772-4cc4-87c8-f9222bdd5488" Sep 08 16:23:46.764869 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.764698 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeReady" Sep 08 16:23:46.764983 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.764972 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 16:23:46.822602 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.822577 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q8bwr"] Sep 08 16:23:46.825055 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.825036 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dm64v"] Sep 08 16:23:46.825157 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.825140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:46.827373 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.827348 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:46.827475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.827377 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 16:23:46.827475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.827405 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gm7zs\"" Sep 08 16:23:46.827604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.827488 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 16:23:46.830349 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.830326 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qgrb7\"" Sep 08 16:23:46.830464 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.830449 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 16:23:46.830575 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.830559 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 16:23:46.830824 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.830811 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 16:23:46.830901 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.830849 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 16:23:46.837408 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.837387 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q8bwr"] Sep 08 16:23:46.840415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.840382 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dm64v"] Sep 08 16:23:46.910010 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.909980 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-metrics-tls\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:46.910010 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.910010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a892e068-3486-42a1-8dff-472b490d9721-crio-socket\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:46.910208 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.910083 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-config-volume\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:46.910208 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.910129 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-tmp-dir\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:46.910208 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.910146 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8kw\" (UniqueName: \"kubernetes.io/projected/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-kube-api-access-8g8kw\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:46.910208 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.910182 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a892e068-3486-42a1-8dff-472b490d9721-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:46.910208 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.910207 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a892e068-3486-42a1-8dff-472b490d9721-data-volume\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:46.910376 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.910224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdl5v\" (UniqueName: \"kubernetes.io/projected/a892e068-3486-42a1-8dff-472b490d9721-kube-api-access-zdl5v\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:46.910376 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.910264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a892e068-3486-42a1-8dff-472b490d9721-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:46.917096 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.917075 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zvhbh"] Sep 08 16:23:46.919290 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.919278 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zvhbh" Sep 08 16:23:46.921674 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.921650 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 16:23:46.921792 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.921689 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 16:23:46.921792 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.921727 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4mdpt\"" Sep 08 16:23:46.921792 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.921771 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 16:23:46.927798 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:46.927771 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zvhbh"] Sep 08 16:23:47.011269 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011235 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-config-volume\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:47.011445 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-tmp-dir\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:47.011445 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011336 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8kw\" (UniqueName: \"kubernetes.io/projected/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-kube-api-access-8g8kw\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:47.011445 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a892e068-3486-42a1-8dff-472b490d9721-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.011445 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011387 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99976398-38d1-4613-891d-cb52ffd9af35-cert\") pod \"ingress-canary-zvhbh\" (UID: \"99976398-38d1-4613-891d-cb52ffd9af35\") " pod="openshift-ingress-canary/ingress-canary-zvhbh" Sep 08 16:23:47.011648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011514 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slgn\" (UniqueName: \"kubernetes.io/projected/99976398-38d1-4613-891d-cb52ffd9af35-kube-api-access-7slgn\") pod \"ingress-canary-zvhbh\" (UID: \"99976398-38d1-4613-891d-cb52ffd9af35\") " pod="openshift-ingress-canary/ingress-canary-zvhbh" Sep 08 16:23:47.011648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a892e068-3486-42a1-8dff-472b490d9721-data-volume\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.011648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011601 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdl5v\" (UniqueName: \"kubernetes.io/projected/a892e068-3486-42a1-8dff-472b490d9721-kube-api-access-zdl5v\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.011648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a892e068-3486-42a1-8dff-472b490d9721-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.011819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-metrics-tls\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:47.011819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011708 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a892e068-3486-42a1-8dff-472b490d9721-crio-socket\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.011910 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a892e068-3486-42a1-8dff-472b490d9721-crio-socket\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.011977 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.011952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-tmp-dir\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:47.012169 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.012146 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a892e068-3486-42a1-8dff-472b490d9721-data-volume\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.012242 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.012224 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-config-volume\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:47.012292 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.012248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a892e068-3486-42a1-8dff-472b490d9721-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.015473 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.015453 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-metrics-tls\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:47.015570 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.015508 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a892e068-3486-42a1-8dff-472b490d9721-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.021682 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.021654 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8kw\" (UniqueName: \"kubernetes.io/projected/f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26-kube-api-access-8g8kw\") pod \"dns-default-q8bwr\" (UID: \"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26\") " pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:47.021757 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.021680 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdl5v\" (UniqueName: \"kubernetes.io/projected/a892e068-3486-42a1-8dff-472b490d9721-kube-api-access-zdl5v\") pod \"insights-runtime-extractor-dm64v\" (UID: \"a892e068-3486-42a1-8dff-472b490d9721\") " pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.046691 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.046664 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:47.049452 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.049430 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7dt42\"" Sep 08 16:23:47.049452 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.049450 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 16:23:47.112530 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.112495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99976398-38d1-4613-891d-cb52ffd9af35-cert\") pod \"ingress-canary-zvhbh\" (UID: \"99976398-38d1-4613-891d-cb52ffd9af35\") " pod="openshift-ingress-canary/ingress-canary-zvhbh" Sep 08 16:23:47.112530 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.112532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slgn\" (UniqueName: \"kubernetes.io/projected/99976398-38d1-4613-891d-cb52ffd9af35-kube-api-access-7slgn\") pod \"ingress-canary-zvhbh\" (UID: \"99976398-38d1-4613-891d-cb52ffd9af35\") " pod="openshift-ingress-canary/ingress-canary-zvhbh" Sep 08 16:23:47.114932 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.114911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99976398-38d1-4613-891d-cb52ffd9af35-cert\") pod \"ingress-canary-zvhbh\" (UID: \"99976398-38d1-4613-891d-cb52ffd9af35\") " pod="openshift-ingress-canary/ingress-canary-zvhbh" Sep 08 16:23:47.120210 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.120189 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slgn\" (UniqueName: \"kubernetes.io/projected/99976398-38d1-4613-891d-cb52ffd9af35-kube-api-access-7slgn\") pod \"ingress-canary-zvhbh\" (UID: \"99976398-38d1-4613-891d-cb52ffd9af35\") " pod="openshift-ingress-canary/ingress-canary-zvhbh" Sep 08 16:23:47.136028 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.136004 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:47.143383 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.143366 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dm64v" Sep 08 16:23:47.227727 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.227702 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zvhbh" Sep 08 16:23:47.420546 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.420517 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q8bwr"] Sep 08 16:23:47.424163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.424136 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dm64v"] Sep 08 16:23:47.424706 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:23:47.424679 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d56ec0_c9ed_44b1_99b3_bfd33db6ee26.slice/crio-58ae6cc8bbf3823461c7b32b53d0f7e316114b9eff5ba2af1c6e5759e3c72216 WatchSource:0}: Error finding container 58ae6cc8bbf3823461c7b32b53d0f7e316114b9eff5ba2af1c6e5759e3c72216: Status 404 returned error can't find the container with id 58ae6cc8bbf3823461c7b32b53d0f7e316114b9eff5ba2af1c6e5759e3c72216 Sep 08 16:23:47.427945 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:23:47.427923 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda892e068_3486_42a1_8dff_472b490d9721.slice/crio-6e5e78833b0c1cd8ac25a0aba36cb1bbd2e9bd649a7990c76ed0e67aa8cf2968 WatchSource:0}: Error finding container 6e5e78833b0c1cd8ac25a0aba36cb1bbd2e9bd649a7990c76ed0e67aa8cf2968: Status 404 returned error can't find the container with id 6e5e78833b0c1cd8ac25a0aba36cb1bbd2e9bd649a7990c76ed0e67aa8cf2968 Sep 08 16:23:47.493845 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:47.493725 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zvhbh"] Sep 08 16:23:47.497623 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:23:47.497600 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99976398_38d1_4613_891d_cb52ffd9af35.slice/crio-51ceedb2681bee76ba096199920648002c503c3932ea7986f249862f1fcaa3b3 WatchSource:0}: Error finding container 51ceedb2681bee76ba096199920648002c503c3932ea7986f249862f1fcaa3b3: Status 404 returned error can't find the container with id 51ceedb2681bee76ba096199920648002c503c3932ea7986f249862f1fcaa3b3 Sep 08 16:23:48.046828 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.046793 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:48.046973 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.046800 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:48.049822 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.049797 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 16:23:48.050919 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.050651 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 16:23:48.050919 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.050668 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cvc8x\"" Sep 08 16:23:48.050919 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.050700 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 16:23:48.333433 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.333275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zvhbh" event={"ID":"99976398-38d1-4613-891d-cb52ffd9af35","Type":"ContainerStarted","Data":"51ceedb2681bee76ba096199920648002c503c3932ea7986f249862f1fcaa3b3"} Sep 08 16:23:48.335969 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.335936 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dm64v" event={"ID":"a892e068-3486-42a1-8dff-472b490d9721","Type":"ContainerStarted","Data":"67b35ae18b01d6db2462d4409ef836c7a23810fbc27d590568ebfda2aa16f923"} Sep 08 16:23:48.336098 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.336001 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dm64v" event={"ID":"a892e068-3486-42a1-8dff-472b490d9721","Type":"ContainerStarted","Data":"6e5e78833b0c1cd8ac25a0aba36cb1bbd2e9bd649a7990c76ed0e67aa8cf2968"} Sep 08 16:23:48.338884 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:48.338805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8bwr" event={"ID":"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26","Type":"ContainerStarted","Data":"58ae6cc8bbf3823461c7b32b53d0f7e316114b9eff5ba2af1c6e5759e3c72216"} Sep 08 16:23:49.343505 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:49.343470 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dm64v" event={"ID":"a892e068-3486-42a1-8dff-472b490d9721","Type":"ContainerStarted","Data":"18a0c893a798f037e4c549d4a374e06f51588d39500f83e20f883b16ba2e8191"} Sep 08 16:23:50.348187 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.348147 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8bwr" event={"ID":"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26","Type":"ContainerStarted","Data":"59a94fb925c75e5018490a40e2ec8e3eca22e3fdb128cc043a0e81fbe36172ca"} Sep 08 16:23:50.348592 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.348214 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8bwr" event={"ID":"f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26","Type":"ContainerStarted","Data":"fe9a61e5654a870ee9e81b5fc6bedbe03cd5dcb9f1455838c070f814a343adc2"} Sep 08 16:23:50.348592 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.348320 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q8bwr" Sep 08 16:23:50.369859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.369807 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q8bwr" podStartSLOduration=2.504754523 podStartE2EDuration="4.369791168s" totalImagesPullingTime="1.865036645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:50.368274019 +0000 UTC m=+57.839260476" watchObservedRunningTime="2026-09-08 16:23:50.369791168 +0000 UTC m=+57.840777623" Sep 08 16:23:50.772823 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.772796 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-thmg6"] Sep 08 16:23:50.775741 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.775719 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-thmg6" Sep 08 16:23:50.778413 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.778382 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 16:23:50.778715 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.778698 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 16:23:50.778851 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.778831 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zmkh7\"" Sep 08 16:23:50.788800 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.788770 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-thmg6"] Sep 08 16:23:50.839956 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.839921 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2qj5\" (UniqueName: \"kubernetes.io/projected/2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed-kube-api-access-f2qj5\") pod \"downloads-85d98c998f-thmg6\" (UID: \"2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed\") " pod="openshift-console/downloads-85d98c998f-thmg6" Sep 08 16:23:50.940436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.940354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2qj5\" (UniqueName: \"kubernetes.io/projected/2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed-kube-api-access-f2qj5\") pod \"downloads-85d98c998f-thmg6\" (UID: \"2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed\") " pod="openshift-console/downloads-85d98c998f-thmg6" Sep 08 16:23:50.948921 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:50.948891 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2qj5\" (UniqueName: \"kubernetes.io/projected/2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed-kube-api-access-f2qj5\") pod \"downloads-85d98c998f-thmg6\" (UID: \"2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed\") " pod="openshift-console/downloads-85d98c998f-thmg6" Sep 08 16:23:51.085419 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:51.085376 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-thmg6" Sep 08 16:23:51.353066 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:51.352958 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zvhbh" event={"ID":"99976398-38d1-4613-891d-cb52ffd9af35","Type":"ContainerStarted","Data":"8b8b5994ca539e397725108cf58659d1f0bae3009a93b9d2bfa3dddf59507e3a"} Sep 08 16:23:51.355527 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:51.355492 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dm64v" event={"ID":"a892e068-3486-42a1-8dff-472b490d9721","Type":"ContainerStarted","Data":"4b06aee7ec10ada220feb765458869e0ee4ef8a1dffb54a943e8af589add13b3"} Sep 08 16:23:51.355725 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:51.355585 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-thmg6"] Sep 08 16:23:51.360385 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:23:51.360364 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e685fc5_c9a0_4df7_bdcb_1ef50eaf3eed.slice/crio-f71ca436a9b21c845abd6d8c29fc9bd177a956aceaf40223815ef54e15e836ab WatchSource:0}: Error finding container f71ca436a9b21c845abd6d8c29fc9bd177a956aceaf40223815ef54e15e836ab: Status 404 returned error can't find the container with id f71ca436a9b21c845abd6d8c29fc9bd177a956aceaf40223815ef54e15e836ab Sep 08 16:23:51.372722 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:51.372585 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zvhbh" podStartSLOduration=2.190869771 podStartE2EDuration="5.37256891s" totalImagesPullingTime="3.181699139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:51.372556225 +0000 UTC m=+58.843542681" watchObservedRunningTime="2026-09-08 16:23:51.37256891 +0000 UTC m=+58.843555372" Sep 08 16:23:51.399347 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:51.399275 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dm64v" podStartSLOduration=2.290458825 podStartE2EDuration="5.39925875s" totalImagesPullingTime="3.108799925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:23:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:51.398533077 +0000 UTC m=+58.869519532" watchObservedRunningTime="2026-09-08 16:23:51.39925875 +0000 UTC m=+58.870245205" Sep 08 16:23:52.359964 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:52.359924 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-thmg6" event={"ID":"2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed","Type":"ContainerStarted","Data":"f71ca436a9b21c845abd6d8c29fc9bd177a956aceaf40223815ef54e15e836ab"} Sep 08 16:23:53.271198 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:53.271167 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vfmw" Sep 08 16:23:55.191135 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.191093 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wbhl9"] Sep 08 16:23:55.194103 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.194081 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.196339 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.196318 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 08 16:23:55.197365 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.197346 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tkkpz\"" Sep 08 16:23:55.197440 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.197350 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 16:23:55.197891 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.197865 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 08 16:23:55.197891 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.197857 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 16:23:55.198059 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.197942 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 16:23:55.205645 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.205586 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wbhl9"] Sep 08 16:23:55.269655 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.269625 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699bz\" (UniqueName: \"kubernetes.io/projected/f4fa655f-11ee-49c0-9f94-914abaf95664-kube-api-access-699bz\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.269807 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.269665 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4fa655f-11ee-49c0-9f94-914abaf95664-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.269807 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.269699 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4fa655f-11ee-49c0-9f94-914abaf95664-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.269807 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.269745 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4fa655f-11ee-49c0-9f94-914abaf95664-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.370239 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.370204 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4fa655f-11ee-49c0-9f94-914abaf95664-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.370414 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.370279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699bz\" (UniqueName: \"kubernetes.io/projected/f4fa655f-11ee-49c0-9f94-914abaf95664-kube-api-access-699bz\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.370414 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.370312 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4fa655f-11ee-49c0-9f94-914abaf95664-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.370414 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.370348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4fa655f-11ee-49c0-9f94-914abaf95664-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.370569 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:55.370459 2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 08 16:23:55.370569 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:23:55.370551 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fa655f-11ee-49c0-9f94-914abaf95664-prometheus-operator-tls podName:f4fa655f-11ee-49c0-9f94-914abaf95664 nodeName:}" failed. No retries permitted until 2026-09-08 16:23:55.870530434 +0000 UTC m=+63.341516891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f4fa655f-11ee-49c0-9f94-914abaf95664-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-wbhl9" (UID: "f4fa655f-11ee-49c0-9f94-914abaf95664") : secret "prometheus-operator-tls" not found Sep 08 16:23:55.372057 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.371396 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4fa655f-11ee-49c0-9f94-914abaf95664-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.375631 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.375605 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4fa655f-11ee-49c0-9f94-914abaf95664-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.379189 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.379163 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-699bz\" (UniqueName: \"kubernetes.io/projected/f4fa655f-11ee-49c0-9f94-914abaf95664-kube-api-access-699bz\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.874343 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.874303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4fa655f-11ee-49c0-9f94-914abaf95664-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:55.877163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:55.877130 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4fa655f-11ee-49c0-9f94-914abaf95664-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wbhl9\" (UID: \"f4fa655f-11ee-49c0-9f94-914abaf95664\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:56.102432 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:56.102395 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" Sep 08 16:23:56.381422 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:56.381390 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wbhl9"] Sep 08 16:23:56.384189 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:23:56.384156 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fa655f_11ee_49c0_9f94_914abaf95664.slice/crio-7393fa7732855a0c32159cdfc06386f6910ea1e1b5c5b2a237ea8e9fcbeb98e7 WatchSource:0}: Error finding container 7393fa7732855a0c32159cdfc06386f6910ea1e1b5c5b2a237ea8e9fcbeb98e7: Status 404 returned error can't find the container with id 7393fa7732855a0c32159cdfc06386f6910ea1e1b5c5b2a237ea8e9fcbeb98e7 Sep 08 16:23:57.376121 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.376081 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" event={"ID":"f4fa655f-11ee-49c0-9f94-914abaf95664","Type":"ContainerStarted","Data":"7393fa7732855a0c32159cdfc06386f6910ea1e1b5c5b2a237ea8e9fcbeb98e7"} Sep 08 16:23:57.686558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.686509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:57.689049 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.689007 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 16:23:57.699758 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.699699 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e8e45e-b772-4cc4-87c8-f9222bdd5488-original-pull-secret\") pod \"global-pull-secret-syncer-cqvbk\" (UID: \"a3e8e45e-b772-4cc4-87c8-f9222bdd5488\") " pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:57.787278 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.787164 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:57.789838 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.789806 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 16:23:57.800799 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.800772 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29-metrics-certs\") pod \"network-metrics-daemon-426b4\" (UID: \"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29\") " pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:57.857909 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.857869 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7dt42\"" Sep 08 16:23:57.865108 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.865084 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-426b4" Sep 08 16:23:57.888422 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.888395 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:57.890961 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.890939 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 16:23:57.900905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.900879 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 16:23:57.911860 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.911815 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sq8s\" (UniqueName: \"kubernetes.io/projected/a4e974fd-6a6f-40b1-a3a4-c213ad566b23-kube-api-access-2sq8s\") pod \"network-check-target-hp42x\" (UID: \"a4e974fd-6a6f-40b1-a3a4-c213ad566b23\") " pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:57.960857 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.960645 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cvc8x\"" Sep 08 16:23:57.965152 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.964944 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqvbk" Sep 08 16:23:57.968822 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:57.968782 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:23:58.159678 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:58.159595 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-426b4"] Sep 08 16:23:58.166653 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:23:58.166375 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6d7cbf_e5f1_476c_ad89_0c9a3b46df29.slice/crio-150e60d7ff12d75499e1cbf8fcd7330247dff88449e2d0a0c1731eb530b3e03a WatchSource:0}: Error finding container 150e60d7ff12d75499e1cbf8fcd7330247dff88449e2d0a0c1731eb530b3e03a: Status 404 returned error can't find the container with id 150e60d7ff12d75499e1cbf8fcd7330247dff88449e2d0a0c1731eb530b3e03a Sep 08 16:23:58.254702 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:58.254628 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cqvbk"] Sep 08 16:23:58.258731 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:58.258707 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hp42x"] Sep 08 16:23:58.259513 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:23:58.259494 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e8e45e_b772_4cc4_87c8_f9222bdd5488.slice/crio-b8c0666279afafaeb17c3e977eca6371a7c3bba4e58470380788abbdab369b81 WatchSource:0}: Error finding container b8c0666279afafaeb17c3e977eca6371a7c3bba4e58470380788abbdab369b81: Status 404 returned error can't find the container with id b8c0666279afafaeb17c3e977eca6371a7c3bba4e58470380788abbdab369b81 Sep 08 16:23:58.261447 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:23:58.261366 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e974fd_6a6f_40b1_a3a4_c213ad566b23.slice/crio-a9dfaedfca36655a1f24d37205a42d831c205c83b84e15090a57e2291f62e68a WatchSource:0}: Error finding container a9dfaedfca36655a1f24d37205a42d831c205c83b84e15090a57e2291f62e68a: Status 404 returned error can't find the container with id a9dfaedfca36655a1f24d37205a42d831c205c83b84e15090a57e2291f62e68a Sep 08 16:23:58.382033 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:58.381979 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" event={"ID":"f4fa655f-11ee-49c0-9f94-914abaf95664","Type":"ContainerStarted","Data":"8a9c22586ce60ebb5f532cd06465456c45b65a7da729d49b3b546f1c06116648"} Sep 08 16:23:58.382259 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:58.382067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" event={"ID":"f4fa655f-11ee-49c0-9f94-914abaf95664","Type":"ContainerStarted","Data":"6b81d7af50f41acfad33cd6e77f64e8cb6dc5ec6e267ac937f8e557e4c2d1dc8"} Sep 08 16:23:58.383615 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:58.383585 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cqvbk" event={"ID":"a3e8e45e-b772-4cc4-87c8-f9222bdd5488","Type":"ContainerStarted","Data":"b8c0666279afafaeb17c3e977eca6371a7c3bba4e58470380788abbdab369b81"} Sep 08 16:23:58.384918 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:58.384889 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hp42x" event={"ID":"a4e974fd-6a6f-40b1-a3a4-c213ad566b23","Type":"ContainerStarted","Data":"a9dfaedfca36655a1f24d37205a42d831c205c83b84e15090a57e2291f62e68a"} Sep 08 16:23:58.386238 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:58.386215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-426b4" event={"ID":"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29","Type":"ContainerStarted","Data":"150e60d7ff12d75499e1cbf8fcd7330247dff88449e2d0a0c1731eb530b3e03a"} Sep 08 16:23:58.399956 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:23:58.399915 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-wbhl9" podStartSLOduration=1.925485454 podStartE2EDuration="3.399897148s" totalImagesPullingTime="1.474411694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:58.399163459 +0000 UTC m=+65.870149916" watchObservedRunningTime="2026-09-08 16:23:58.399897148 +0000 UTC m=+65.870883606" Sep 08 16:24:00.358495 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.358460 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q8bwr" Sep 08 16:24:00.400409 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.400373 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-426b4" event={"ID":"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29","Type":"ContainerStarted","Data":"181cd4bdbb38fef5a29581d2b4347568bdad7011e93095f2a6c496d3897a4aa8"} Sep 08 16:24:00.400603 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.400442 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-426b4" event={"ID":"bb6d7cbf-e5f1-476c-ad89-0c9a3b46df29","Type":"ContainerStarted","Data":"8ec8afb535a7d41cba0c186d178c3d2a5a82ed31fa19f5e02f17a8d9ff0948da"} Sep 08 16:24:00.558338 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.558275 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-426b4" podStartSLOduration=66.302239043 podStartE2EDuration="1m7.558250139s" totalImagesPullingTime="1.256011096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:00.417585049 +0000 UTC m=+67.888571498" watchObservedRunningTime="2026-09-08 16:24:00.558250139 +0000 UTC m=+68.029236594" Sep 08 16:24:00.559451 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.559425 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4"] Sep 08 16:24:00.563252 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.563234 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.566104 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.566081 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 08 16:24:00.566211 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.566094 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 08 16:24:00.569196 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.567636 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-phqd7\"" Sep 08 16:24:00.578514 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.578418 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4"] Sep 08 16:24:00.584126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.581034 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gc886"] Sep 08 16:24:00.585379 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.585362 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-58zzz"] Sep 08 16:24:00.585711 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.585690 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.588227 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.588207 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 08 16:24:00.588293 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.588224 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jxdfc\"" Sep 08 16:24:00.588377 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.588359 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.588899 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.588885 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 08 16:24:00.589166 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.589151 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 08 16:24:00.594129 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.593065 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 16:24:00.594129 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.593085 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 16:24:00.594129 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.593357 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 16:24:00.594129 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.594123 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vt2q9\"" Sep 08 16:24:00.599898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.599798 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gc886"] Sep 08 16:24:00.611106 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611053 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.611179 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611104 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.611179 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611133 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97526267-ea08-4c82-aa89-5aeee4646510-metrics-client-ca\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.611179 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611157 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97526267-ea08-4c82-aa89-5aeee4646510-root\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.611297 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611184 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7t49\" (UniqueName: \"kubernetes.io/projected/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-kube-api-access-z7t49\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.611297 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611213 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.611297 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611244 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c92f55be-0da5-4c92-81bf-93ae92a68386-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.611297 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611273 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-textfile\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.611433 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611298 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-accelerators-collector-config\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.611433 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611322 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrl7v\" (UniqueName: \"kubernetes.io/projected/97526267-ea08-4c82-aa89-5aeee4646510-kube-api-access-jrl7v\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.611433 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611355 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4jfm\" (UniqueName: \"kubernetes.io/projected/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-api-access-q4jfm\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.611433 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611383 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97526267-ea08-4c82-aa89-5aeee4646510-sys\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.611433 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.611605 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611434 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.611605 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611465 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.611605 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611494 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-wtmp\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.611605 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611535 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.611605 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611561 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c92f55be-0da5-4c92-81bf-93ae92a68386-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.611605 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.611584 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-tls\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.712696 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.712662 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.712696 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.712708 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c92f55be-0da5-4c92-81bf-93ae92a68386-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.712933 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.712878 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-tls\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.712933 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.712922 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.713142 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.712963 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.713142 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.712994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97526267-ea08-4c82-aa89-5aeee4646510-metrics-client-ca\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713142 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:00.713049 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 08 16:24:00.713142 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713104 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97526267-ea08-4c82-aa89-5aeee4646510-root\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713142 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:00.713123 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-tls podName:97526267-ea08-4c82-aa89-5aeee4646510 nodeName:}" failed. No retries permitted until 2026-09-08 16:24:01.213099181 +0000 UTC m=+68.684085636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-tls") pod "node-exporter-58zzz" (UID: "97526267-ea08-4c82-aa89-5aeee4646510") : secret "node-exporter-tls" not found Sep 08 16:24:00.713142 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97526267-ea08-4c82-aa89-5aeee4646510-root\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713142 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713136 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c92f55be-0da5-4c92-81bf-93ae92a68386-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7t49\" (UniqueName: \"kubernetes.io/projected/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-kube-api-access-z7t49\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713202 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c92f55be-0da5-4c92-81bf-93ae92a68386-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713269 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-textfile\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713292 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-accelerators-collector-config\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrl7v\" (UniqueName: \"kubernetes.io/projected/97526267-ea08-4c82-aa89-5aeee4646510-kube-api-access-jrl7v\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4jfm\" (UniqueName: \"kubernetes.io/projected/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-api-access-q4jfm\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97526267-ea08-4c82-aa89-5aeee4646510-sys\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713446 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.713481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713479 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.713996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-wtmp\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97526267-ea08-4c82-aa89-5aeee4646510-sys\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713630 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-textfile\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713680 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-wtmp\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.713996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713733 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97526267-ea08-4c82-aa89-5aeee4646510-metrics-client-ca\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.713996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.713726 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.713996 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:00.713806 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 08 16:24:00.713996 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:00.713862 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-openshift-state-metrics-tls podName:bfb80ebb-1763-4fa4-9d57-3bb46df66d79 nodeName:}" failed. No retries permitted until 2026-09-08 16:24:01.213844342 +0000 UTC m=+68.684830799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-xs9n4" (UID: "bfb80ebb-1763-4fa4-9d57-3bb46df66d79") : secret "openshift-state-metrics-tls" not found Sep 08 16:24:00.714328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.714156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-accelerators-collector-config\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.714367 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.714353 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c92f55be-0da5-4c92-81bf-93ae92a68386-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.717076 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.717011 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.717223 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.717198 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.718700 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.718664 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.719978 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.719959 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.725728 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.725700 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7t49\" (UniqueName: \"kubernetes.io/projected/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-kube-api-access-z7t49\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:00.726815 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.726788 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4jfm\" (UniqueName: \"kubernetes.io/projected/c92f55be-0da5-4c92-81bf-93ae92a68386-kube-api-access-q4jfm\") pod \"kube-state-metrics-74cb57df7-gc886\" (UID: \"c92f55be-0da5-4c92-81bf-93ae92a68386\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:00.726918 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.726853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrl7v\" (UniqueName: \"kubernetes.io/projected/97526267-ea08-4c82-aa89-5aeee4646510-kube-api-access-jrl7v\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:00.901352 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:00.901096 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" Sep 08 16:24:01.208909 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.208877 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gc886"] Sep 08 16:24:01.218421 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.218383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:01.218561 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.218447 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-tls\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:01.221752 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.221728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfb80ebb-1763-4fa4-9d57-3bb46df66d79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xs9n4\" (UID: \"bfb80ebb-1763-4fa4-9d57-3bb46df66d79\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:01.223646 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.223621 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97526267-ea08-4c82-aa89-5aeee4646510-node-exporter-tls\") pod \"node-exporter-58zzz\" (UID: \"97526267-ea08-4c82-aa89-5aeee4646510\") " pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:01.303759 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.303725 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84bfdffcd8-k7frl"] Sep 08 16:24:01.308412 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.308385 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.310899 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.310873 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 16:24:01.311082 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.311007 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 16:24:01.311158 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.311112 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 16:24:01.311224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.311176 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 16:24:01.311277 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.311252 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p2kn4\"" Sep 08 16:24:01.311330 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.311316 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 16:24:01.316886 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.316864 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84bfdffcd8-k7frl"] Sep 08 16:24:01.420425 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.420098 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-oauth-config\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.420425 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.420150 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-serving-cert\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.420425 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.420189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-service-ca\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.420425 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.420220 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6cd4\" (UniqueName: \"kubernetes.io/projected/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-kube-api-access-s6cd4\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.420425 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.420247 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-oauth-serving-cert\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.420425 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.420304 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-config\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.479175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.479096 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" Sep 08 16:24:01.509502 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.509468 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-58zzz" Sep 08 16:24:01.521762 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.521725 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-config\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.521982 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.521889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-oauth-config\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.521982 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.521965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-serving-cert\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.522140 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.522006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-service-ca\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.522140 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.522066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6cd4\" (UniqueName: \"kubernetes.io/projected/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-kube-api-access-s6cd4\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.522140 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.522098 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-oauth-serving-cert\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.522802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.522471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-config\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.522802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.522756 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-oauth-serving-cert\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.523092 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.523064 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-service-ca\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.524974 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.524952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-serving-cert\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.525305 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.525284 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-oauth-config\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.534050 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.533999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6cd4\" (UniqueName: \"kubernetes.io/projected/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-kube-api-access-s6cd4\") pod \"console-84bfdffcd8-k7frl\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.620980 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.620939 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:01.644544 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.644514 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:24:01.648928 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.648373 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.650897 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.650872 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 08 16:24:01.651327 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.651139 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 08 16:24:01.651327 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.651177 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 08 16:24:01.651327 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.651202 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 08 16:24:01.651327 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.651230 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 08 16:24:01.651327 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.651267 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c7gwg\"" Sep 08 16:24:01.651721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.651500 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 08 16:24:01.651721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.651538 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 08 16:24:01.651721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.651660 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 08 16:24:01.651870 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.651793 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 08 16:24:01.663417 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.663332 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:24:01.723174 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723137 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgl9p\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-kube-api-access-xgl9p\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723213 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723233 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723252 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-web-config\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723273 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-config-out\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723291 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723307 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723323 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723368 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-config-volume\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723386 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723424 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.723458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.723445 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824316 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-web-config\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824316 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824285 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-config-out\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824316 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824313 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824369 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-config-volume\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824440 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824464 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824480 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824500 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgl9p\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-kube-api-access-xgl9p\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824518 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824538 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.824587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.824552 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.825606 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:01.825293 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-trusted-ca-bundle podName:1695e5ed-a7fd-412d-a954-650bbc051e90 nodeName:}" failed. No retries permitted until 2026-09-08 16:24:02.325268039 +0000 UTC m=+69.796254475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90") : configmap references non-existent config key: ca-bundle.crt Sep 08 16:24:01.825606 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:01.825436 2567 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 08 16:24:01.825793 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:01.825654 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-main-tls podName:1695e5ed-a7fd-412d-a954-650bbc051e90 nodeName:}" failed. No retries permitted until 2026-09-08 16:24:02.325567331 +0000 UTC m=+69.796553767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90") : secret "alertmanager-main-tls" not found Sep 08 16:24:01.825793 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.825739 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.827010 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.826533 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.828080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.827991 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.828820 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.828783 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-web-config\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.829492 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.829449 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.830133 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.830091 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.830380 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.830326 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.831677 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.831655 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-config-volume\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.831761 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.831734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-config-out\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.832577 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.832552 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:01.838407 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:01.838362 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgl9p\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-kube-api-access-xgl9p\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:02.329420 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:02.329378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:02.329603 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:02.329430 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:02.330362 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:02.330336 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:02.332235 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:02.332206 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:02.562282 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:02.562243 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:24:03.557794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.557755 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d88d5644f-x68ct"] Sep 08 16:24:03.561896 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.561869 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.565652 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.565623 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 08 16:24:03.566061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.565734 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fdu7k2llk44ue\"" Sep 08 16:24:03.566061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.565770 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 08 16:24:03.566061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.565986 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 08 16:24:03.566061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.565987 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xcgcr\"" Sep 08 16:24:03.566855 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.566506 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 08 16:24:03.566855 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.566705 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 08 16:24:03.576567 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.576538 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d88d5644f-x68ct"] Sep 08 16:24:03.641475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.641185 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f517b143-d4bb-496d-9838-c554d4babd5b-metrics-client-ca\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.641475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.641241 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.641475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.641292 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkcx5\" (UniqueName: \"kubernetes.io/projected/f517b143-d4bb-496d-9838-c554d4babd5b-kube-api-access-bkcx5\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.641475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.641328 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-tls\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.641475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.641359 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-grpc-tls\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.641475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.641389 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.641475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.641416 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.641954 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.641474 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.742230 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.742187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.742415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.742265 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkcx5\" (UniqueName: \"kubernetes.io/projected/f517b143-d4bb-496d-9838-c554d4babd5b-kube-api-access-bkcx5\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.742415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.742305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-tls\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.742415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.742337 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-grpc-tls\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.742415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.742367 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.742415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.742397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.742624 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.742425 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.742624 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.742471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f517b143-d4bb-496d-9838-c554d4babd5b-metrics-client-ca\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.743760 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.743701 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f517b143-d4bb-496d-9838-c554d4babd5b-metrics-client-ca\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.746115 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.746089 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.746228 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.746189 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-tls\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.763598 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.746802 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.763598 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.747158 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.763598 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.747371 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-grpc-tls\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.763598 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.748312 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f517b143-d4bb-496d-9838-c554d4babd5b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.763598 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.750754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkcx5\" (UniqueName: \"kubernetes.io/projected/f517b143-d4bb-496d-9838-c554d4babd5b-kube-api-access-bkcx5\") pod \"thanos-querier-6d88d5644f-x68ct\" (UID: \"f517b143-d4bb-496d-9838-c554d4babd5b\") " pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:03.797321 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:03.797280 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92f55be_0da5_4c92_81bf_93ae92a68386.slice/crio-cbeaff29ac16c99d1889126b1ef2c83e7c4a8bddbfc3c854c868d09fb9030eb4 WatchSource:0}: Error finding container cbeaff29ac16c99d1889126b1ef2c83e7c4a8bddbfc3c854c868d09fb9030eb4: Status 404 returned error can't find the container with id cbeaff29ac16c99d1889126b1ef2c83e7c4a8bddbfc3c854c868d09fb9030eb4 Sep 08 16:24:03.865004 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:03.864971 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97526267_ea08_4c82_aa89_5aeee4646510.slice/crio-c36e41e63953b909cda994bd6e6ef6c7663e97909b81ba0a42fa3e219438e078 WatchSource:0}: Error finding container c36e41e63953b909cda994bd6e6ef6c7663e97909b81ba0a42fa3e219438e078: Status 404 returned error can't find the container with id c36e41e63953b909cda994bd6e6ef6c7663e97909b81ba0a42fa3e219438e078 Sep 08 16:24:03.879399 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:03.879368 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:04.151648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.151565 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84bfdffcd8-k7frl"] Sep 08 16:24:04.154495 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:04.154464 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b41b504_d2a7_437b_ba03_f55ef7a2a9a7.slice/crio-51b655c576a2e77881b3bd8bb39c027b629e8db6102542d40526dda5132ed89c WatchSource:0}: Error finding container 51b655c576a2e77881b3bd8bb39c027b629e8db6102542d40526dda5132ed89c: Status 404 returned error can't find the container with id 51b655c576a2e77881b3bd8bb39c027b629e8db6102542d40526dda5132ed89c Sep 08 16:24:04.179355 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.179271 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4"] Sep 08 16:24:04.182313 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:04.182281 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb80ebb_1763_4fa4_9d57_3bb46df66d79.slice/crio-0a9b625cccc8c84897c041168475ee3736367e2ee538a9af7a6dd65c1aa63457 WatchSource:0}: Error finding container 0a9b625cccc8c84897c041168475ee3736367e2ee538a9af7a6dd65c1aa63457: Status 404 returned error can't find the container with id 0a9b625cccc8c84897c041168475ee3736367e2ee538a9af7a6dd65c1aa63457 Sep 08 16:24:04.184998 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.184879 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:24:04.188256 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:04.188227 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1695e5ed_a7fd_412d_a954_650bbc051e90.slice/crio-385e686f6dd19a95f8626e21118d04a9563b54ca3dbd342b6ff2062d7e6a4de9 WatchSource:0}: Error finding container 385e686f6dd19a95f8626e21118d04a9563b54ca3dbd342b6ff2062d7e6a4de9: Status 404 returned error can't find the container with id 385e686f6dd19a95f8626e21118d04a9563b54ca3dbd342b6ff2062d7e6a4de9 Sep 08 16:24:04.210268 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.210234 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d88d5644f-x68ct"] Sep 08 16:24:04.216488 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:04.216338 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf517b143_d4bb_496d_9838_c554d4babd5b.slice/crio-4519abeb42e4346d63c7ddd0cc60f0b9581598de3140bcc4fb7b2bd40f9c7807 WatchSource:0}: Error finding container 4519abeb42e4346d63c7ddd0cc60f0b9581598de3140bcc4fb7b2bd40f9c7807: Status 404 returned error can't find the container with id 4519abeb42e4346d63c7ddd0cc60f0b9581598de3140bcc4fb7b2bd40f9c7807 Sep 08 16:24:04.420666 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.420621 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" event={"ID":"bfb80ebb-1763-4fa4-9d57-3bb46df66d79","Type":"ContainerStarted","Data":"a8824f7aa5076fb4d3457763f3577e8d2bf71f9975d9f6c6b1ca448ea46731f9"} Sep 08 16:24:04.420829 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.420699 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" event={"ID":"bfb80ebb-1763-4fa4-9d57-3bb46df66d79","Type":"ContainerStarted","Data":"b21050529f0840a2545c2a96d74d5522197dee60b970e9ce3117b3f220582f49"} Sep 08 16:24:04.420829 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.420734 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" event={"ID":"bfb80ebb-1763-4fa4-9d57-3bb46df66d79","Type":"ContainerStarted","Data":"0a9b625cccc8c84897c041168475ee3736367e2ee538a9af7a6dd65c1aa63457"} Sep 08 16:24:04.423400 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.423232 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" event={"ID":"f517b143-d4bb-496d-9838-c554d4babd5b","Type":"ContainerStarted","Data":"4519abeb42e4346d63c7ddd0cc60f0b9581598de3140bcc4fb7b2bd40f9c7807"} Sep 08 16:24:04.425228 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.425198 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cqvbk" event={"ID":"a3e8e45e-b772-4cc4-87c8-f9222bdd5488","Type":"ContainerStarted","Data":"e2251290bac603a7369569c88b495d0be3ae0cb766cce6f0f850dcbc469b2e2e"} Sep 08 16:24:04.427649 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.427534 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerStarted","Data":"385e686f6dd19a95f8626e21118d04a9563b54ca3dbd342b6ff2062d7e6a4de9"} Sep 08 16:24:04.429455 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.429427 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bfdffcd8-k7frl" event={"ID":"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7","Type":"ContainerStarted","Data":"51b655c576a2e77881b3bd8bb39c027b629e8db6102542d40526dda5132ed89c"} Sep 08 16:24:04.431381 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.431356 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-58zzz" event={"ID":"97526267-ea08-4c82-aa89-5aeee4646510","Type":"ContainerStarted","Data":"c36e41e63953b909cda994bd6e6ef6c7663e97909b81ba0a42fa3e219438e078"} Sep 08 16:24:04.432884 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.432850 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" event={"ID":"c92f55be-0da5-4c92-81bf-93ae92a68386","Type":"ContainerStarted","Data":"cbeaff29ac16c99d1889126b1ef2c83e7c4a8bddbfc3c854c868d09fb9030eb4"} Sep 08 16:24:04.435558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.435527 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hp42x" event={"ID":"a4e974fd-6a6f-40b1-a3a4-c213ad566b23","Type":"ContainerStarted","Data":"69568fc29a2ee5ebd3e3e2a332725399fc14212dfcc22dbc9d92d4ab94e083ca"} Sep 08 16:24:04.436111 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.436091 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:24:04.441737 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.441695 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cqvbk" podStartSLOduration=65.829554606 podStartE2EDuration="1m11.441667512s" totalImagesPullingTime="5.612112906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:04.439972393 +0000 UTC m=+71.910958850" watchObservedRunningTime="2026-09-08 16:24:04.441667512 +0000 UTC m=+71.912653974" Sep 08 16:24:04.455564 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.455511 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hp42x" podStartSLOduration=65.853095072 podStartE2EDuration="1m11.455489033s" totalImagesPullingTime="5.602393961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:04.45449267 +0000 UTC m=+71.925479125" watchObservedRunningTime="2026-09-08 16:24:04.455489033 +0000 UTC m=+71.926475496" Sep 08 16:24:04.977030 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.976986 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7fc5949489-w2wk8"] Sep 08 16:24:04.991170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.991138 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fc5949489-w2wk8"] Sep 08 16:24:04.991341 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.991273 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:04.993931 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.993908 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 08 16:24:04.994114 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.993947 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 08 16:24:04.994114 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.993983 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 08 16:24:04.994114 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.994098 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 08 16:24:04.994896 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.994873 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5dp8r\"" Sep 08 16:24:04.995182 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:04.995164 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dh4i05lui5uc8\"" Sep 08 16:24:05.053808 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.053774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a335b4f2-2d52-4f21-95ad-c4f76259c0db-client-ca-bundle\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.053954 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.053819 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a335b4f2-2d52-4f21-95ad-c4f76259c0db-metrics-server-audit-profiles\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.053954 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.053923 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv42k\" (UniqueName: \"kubernetes.io/projected/a335b4f2-2d52-4f21-95ad-c4f76259c0db-kube-api-access-gv42k\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.054081 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.053991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a335b4f2-2d52-4f21-95ad-c4f76259c0db-secret-metrics-server-tls\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.054081 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.054062 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a335b4f2-2d52-4f21-95ad-c4f76259c0db-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.054176 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.054094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a335b4f2-2d52-4f21-95ad-c4f76259c0db-audit-log\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.054176 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.054127 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a335b4f2-2d52-4f21-95ad-c4f76259c0db-secret-metrics-server-client-certs\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.154786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.154748 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a335b4f2-2d52-4f21-95ad-c4f76259c0db-audit-log\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.154786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.154792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a335b4f2-2d52-4f21-95ad-c4f76259c0db-secret-metrics-server-client-certs\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.154999 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.154827 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a335b4f2-2d52-4f21-95ad-c4f76259c0db-client-ca-bundle\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.154999 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.154855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a335b4f2-2d52-4f21-95ad-c4f76259c0db-metrics-server-audit-profiles\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.154999 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.154896 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv42k\" (UniqueName: \"kubernetes.io/projected/a335b4f2-2d52-4f21-95ad-c4f76259c0db-kube-api-access-gv42k\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.154999 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.154942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a335b4f2-2d52-4f21-95ad-c4f76259c0db-secret-metrics-server-tls\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.154999 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.154976 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a335b4f2-2d52-4f21-95ad-c4f76259c0db-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.155278 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.155249 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a335b4f2-2d52-4f21-95ad-c4f76259c0db-audit-log\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.155704 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.155676 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a335b4f2-2d52-4f21-95ad-c4f76259c0db-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.155942 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.155923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a335b4f2-2d52-4f21-95ad-c4f76259c0db-metrics-server-audit-profiles\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.158009 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.157982 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a335b4f2-2d52-4f21-95ad-c4f76259c0db-client-ca-bundle\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.158450 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.158424 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a335b4f2-2d52-4f21-95ad-c4f76259c0db-secret-metrics-server-client-certs\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.158537 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.158449 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a335b4f2-2d52-4f21-95ad-c4f76259c0db-secret-metrics-server-tls\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.164328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.164305 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv42k\" (UniqueName: \"kubernetes.io/projected/a335b4f2-2d52-4f21-95ad-c4f76259c0db-kube-api-access-gv42k\") pod \"metrics-server-7fc5949489-w2wk8\" (UID: \"a335b4f2-2d52-4f21-95ad-c4f76259c0db\") " pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.305092 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.305003 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:05.339394 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.339363 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm"] Sep 08 16:24:05.360221 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.360180 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm"] Sep 08 16:24:05.360377 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.360238 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:05.362669 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.362628 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 08 16:24:05.362823 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.362809 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v6znf\"" Sep 08 16:24:05.458561 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.458527 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614b8d36-dbe9-49fb-b0c0-5da6e73987ea-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-sqqpm\" (UID: \"614b8d36-dbe9-49fb-b0c0-5da6e73987ea\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:05.458719 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.458623 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkg2\" (UniqueName: \"kubernetes.io/projected/614b8d36-dbe9-49fb-b0c0-5da6e73987ea-kube-api-access-lhkg2\") pod \"monitoring-plugin-85bbbbbf4-sqqpm\" (UID: \"614b8d36-dbe9-49fb-b0c0-5da6e73987ea\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:05.560990 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.559866 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614b8d36-dbe9-49fb-b0c0-5da6e73987ea-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-sqqpm\" (UID: \"614b8d36-dbe9-49fb-b0c0-5da6e73987ea\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:05.560990 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.559961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkg2\" (UniqueName: \"kubernetes.io/projected/614b8d36-dbe9-49fb-b0c0-5da6e73987ea-kube-api-access-lhkg2\") pod \"monitoring-plugin-85bbbbbf4-sqqpm\" (UID: \"614b8d36-dbe9-49fb-b0c0-5da6e73987ea\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:05.560990 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:05.560586 2567 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 08 16:24:05.560990 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:05.560649 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/614b8d36-dbe9-49fb-b0c0-5da6e73987ea-monitoring-plugin-cert podName:614b8d36-dbe9-49fb-b0c0-5da6e73987ea nodeName:}" failed. No retries permitted until 2026-09-08 16:24:06.060627569 +0000 UTC m=+73.531614008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/614b8d36-dbe9-49fb-b0c0-5da6e73987ea-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-sqqpm" (UID: "614b8d36-dbe9-49fb-b0c0-5da6e73987ea") : secret "monitoring-plugin-cert" not found Sep 08 16:24:05.572813 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.572653 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkg2\" (UniqueName: \"kubernetes.io/projected/614b8d36-dbe9-49fb-b0c0-5da6e73987ea-kube-api-access-lhkg2\") pod \"monitoring-plugin-85bbbbbf4-sqqpm\" (UID: \"614b8d36-dbe9-49fb-b0c0-5da6e73987ea\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:05.796491 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.795573 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d74d56774-hkgxr"] Sep 08 16:24:05.799418 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.799395 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.803758 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.803548 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 08 16:24:05.803758 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.803664 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 08 16:24:05.805296 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.804887 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 08 16:24:05.805296 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.805002 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lfc8n\"" Sep 08 16:24:05.805296 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.805129 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 08 16:24:05.805753 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.805574 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 08 16:24:05.810362 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.810343 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 08 16:24:05.814975 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.814873 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d74d56774-hkgxr"] Sep 08 16:24:05.864177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.863717 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-telemeter-client-tls\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.864177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.863773 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913426d9-bf7f-443e-8da2-d68e6b953e1f-serving-certs-ca-bundle\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.864177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.863814 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.864177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.863849 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913426d9-bf7f-443e-8da2-d68e6b953e1f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.864177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.863880 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvgw\" (UniqueName: \"kubernetes.io/projected/913426d9-bf7f-443e-8da2-d68e6b953e1f-kube-api-access-qvvgw\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.864177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.863929 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-federate-client-tls\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.864177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.863975 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-secret-telemeter-client\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.864177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.863998 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/913426d9-bf7f-443e-8da2-d68e6b953e1f-metrics-client-ca\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.965366 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.965262 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-secret-telemeter-client\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.965366 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.965307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/913426d9-bf7f-443e-8da2-d68e6b953e1f-metrics-client-ca\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.965545 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.965396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-telemeter-client-tls\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.965545 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.965428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913426d9-bf7f-443e-8da2-d68e6b953e1f-serving-certs-ca-bundle\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.965545 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.965466 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.965545 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.965498 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913426d9-bf7f-443e-8da2-d68e6b953e1f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.965545 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.965527 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvgw\" (UniqueName: \"kubernetes.io/projected/913426d9-bf7f-443e-8da2-d68e6b953e1f-kube-api-access-qvvgw\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.965721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.965570 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-federate-client-tls\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.967607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.966710 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913426d9-bf7f-443e-8da2-d68e6b953e1f-serving-certs-ca-bundle\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.967607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.967559 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/913426d9-bf7f-443e-8da2-d68e6b953e1f-metrics-client-ca\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.967925 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.967910 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913426d9-bf7f-443e-8da2-d68e6b953e1f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.976443 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.976414 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.979292 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.979141 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-federate-client-tls\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.988371 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.988310 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-secret-telemeter-client\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.988462 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.988370 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/913426d9-bf7f-443e-8da2-d68e6b953e1f-telemeter-client-tls\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:05.990126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:05.990092 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvgw\" (UniqueName: \"kubernetes.io/projected/913426d9-bf7f-443e-8da2-d68e6b953e1f-kube-api-access-qvvgw\") pod \"telemeter-client-d74d56774-hkgxr\" (UID: \"913426d9-bf7f-443e-8da2-d68e6b953e1f\") " pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:06.067785 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.067079 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614b8d36-dbe9-49fb-b0c0-5da6e73987ea-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-sqqpm\" (UID: \"614b8d36-dbe9-49fb-b0c0-5da6e73987ea\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:06.070867 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.070819 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/614b8d36-dbe9-49fb-b0c0-5da6e73987ea-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-sqqpm\" (UID: \"614b8d36-dbe9-49fb-b0c0-5da6e73987ea\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:06.115327 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.115296 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" Sep 08 16:24:06.233389 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.233318 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fc5949489-w2wk8"] Sep 08 16:24:06.271839 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.271586 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:06.448241 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.447433 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-58zzz" event={"ID":"97526267-ea08-4c82-aa89-5aeee4646510","Type":"ContainerStarted","Data":"8612cc007b1be9a587a070d9f0d71c8fbdc4e611b04868b7dae16493c232ba62"} Sep 08 16:24:06.450572 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.450508 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" event={"ID":"c92f55be-0da5-4c92-81bf-93ae92a68386","Type":"ContainerStarted","Data":"6d3d6663e1583a43da79f1e58c9519853b1fd2c72681f710bc1c9cfbc46ff5c8"} Sep 08 16:24:06.817794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.817708 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:24:06.823561 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.821976 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.825242 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.825215 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 08 16:24:06.825452 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.825434 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 08 16:24:06.825667 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.825621 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 08 16:24:06.825864 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.825848 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ecml97ua1ifr7\"" Sep 08 16:24:06.826124 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.826106 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ww6bp\"" Sep 08 16:24:06.826866 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.826844 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 08 16:24:06.827248 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.827224 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 08 16:24:06.827531 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.827503 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 08 16:24:06.827615 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.827544 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 08 16:24:06.828642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.827772 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 08 16:24:06.828642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.828148 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 08 16:24:06.828642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.828199 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 08 16:24:06.828642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.828216 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 08 16:24:06.828642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.828150 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 08 16:24:06.828642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.828475 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 08 16:24:06.831011 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.830969 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 08 16:24:06.839389 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.839295 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:24:06.875503 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875399 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.875503 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875452 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.875725 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875551 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.875725 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.875725 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875621 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.875889 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.875889 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875791 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjqk9\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-kube-api-access-pjqk9\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.875889 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875863 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-config-out\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.875889 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875886 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.875941 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.876000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.876062 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.876093 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.876117 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-web-config\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876412 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.876158 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-config\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876412 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.876219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876412 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.876248 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876412 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.876275 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.876412 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.876308 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977222 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977191 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977339 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977444 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977511 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977562 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977601 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977564 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977605 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977699 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977654 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjqk9\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-kube-api-access-pjqk9\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977699 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-config-out\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977792 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977792 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977742 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977792 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977768 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977934 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977799 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977934 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977827 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977934 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977852 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977934 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977876 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-web-config\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.977934 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977915 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.978128 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-config\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.978128 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.977992 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.978128 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.978052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.978128 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.978086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.978312 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.978259 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.979140 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.978934 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.979140 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.979077 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.980289 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.979646 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.980849 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.980826 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.981257 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.981234 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.982303 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.981773 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.982303 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.982262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.982447 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.982392 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.982929 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.982890 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.986490 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.986427 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.986576 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.986504 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-config-out\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.986648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.986589 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.987093 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.986973 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjqk9\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-kube-api-access-pjqk9\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.987093 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.987048 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.987336 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.987297 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-web-config\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.987640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.987580 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.987864 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.987843 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-config\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:06.989150 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:06.989130 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:07.142770 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:07.142495 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:07.460446 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:07.460409 2567 generic.go:356] "Generic (PLEG): container finished" podID="97526267-ea08-4c82-aa89-5aeee4646510" containerID="8612cc007b1be9a587a070d9f0d71c8fbdc4e611b04868b7dae16493c232ba62" exitCode=0 Sep 08 16:24:07.460607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:07.460499 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-58zzz" event={"ID":"97526267-ea08-4c82-aa89-5aeee4646510","Type":"ContainerDied","Data":"8612cc007b1be9a587a070d9f0d71c8fbdc4e611b04868b7dae16493c232ba62"} Sep 08 16:24:07.462319 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:07.462288 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" event={"ID":"a335b4f2-2d52-4f21-95ad-c4f76259c0db","Type":"ContainerStarted","Data":"7136a0509526d5a415b927471a17c498d6143812dd6cdd4e733044ed5e66dc14"} Sep 08 16:24:15.481211 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:15.481175 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm"] Sep 08 16:24:16.418949 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:16.418906 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614b8d36_dbe9_49fb_b0c0_5da6e73987ea.slice/crio-e37276674ea69a2bb6d5d2dca7ebf6503e3afcc7bac46ba4694077245addb749 WatchSource:0}: Error finding container e37276674ea69a2bb6d5d2dca7ebf6503e3afcc7bac46ba4694077245addb749: Status 404 returned error can't find the container with id e37276674ea69a2bb6d5d2dca7ebf6503e3afcc7bac46ba4694077245addb749 Sep 08 16:24:16.502627 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:16.502593 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" event={"ID":"614b8d36-dbe9-49fb-b0c0-5da6e73987ea","Type":"ContainerStarted","Data":"e37276674ea69a2bb6d5d2dca7ebf6503e3afcc7bac46ba4694077245addb749"} Sep 08 16:24:16.706510 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:16.706482 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d74d56774-hkgxr"] Sep 08 16:24:16.710461 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:16.710369 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:24:16.711583 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:16.711550 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913426d9_bf7f_443e_8da2_d68e6b953e1f.slice/crio-c80a8048e649c200363a9df97f5d7ee94c532f9ac9df8289f4a18e4211899852 WatchSource:0}: Error finding container c80a8048e649c200363a9df97f5d7ee94c532f9ac9df8289f4a18e4211899852: Status 404 returned error can't find the container with id c80a8048e649c200363a9df97f5d7ee94c532f9ac9df8289f4a18e4211899852 Sep 08 16:24:17.093513 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:17.093440 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68654cd_4b38_4118_87d9_537792259b79.slice/crio-da248b00256c7aa759ac13c69de9b82f5852b65bbdd58ac6262f9ceccfa65348 WatchSource:0}: Error finding container da248b00256c7aa759ac13c69de9b82f5852b65bbdd58ac6262f9ceccfa65348: Status 404 returned error can't find the container with id da248b00256c7aa759ac13c69de9b82f5852b65bbdd58ac6262f9ceccfa65348 Sep 08 16:24:17.526732 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.522631 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" event={"ID":"c92f55be-0da5-4c92-81bf-93ae92a68386","Type":"ContainerStarted","Data":"ff77d6adff1832c456e68ac23353bf2ba3ad7fd08835d02cfec1b7cd90fcd3b5"} Sep 08 16:24:17.526732 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.522742 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" event={"ID":"c92f55be-0da5-4c92-81bf-93ae92a68386","Type":"ContainerStarted","Data":"08404cf98ed84069d279ecd9f30e90c798f6144d9df94186353c2bad806d32a4"} Sep 08 16:24:17.526732 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.525813 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" event={"ID":"a335b4f2-2d52-4f21-95ad-c4f76259c0db","Type":"ContainerStarted","Data":"6abdf4fd19fe18d37a94486a0c62ec6f59b2c4f2e058db73a124de35a403ed4b"} Sep 08 16:24:17.529606 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.529580 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" event={"ID":"913426d9-bf7f-443e-8da2-d68e6b953e1f","Type":"ContainerStarted","Data":"c80a8048e649c200363a9df97f5d7ee94c532f9ac9df8289f4a18e4211899852"} Sep 08 16:24:17.533592 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.533566 2567 generic.go:356] "Generic (PLEG): container finished" podID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerID="62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612" exitCode=0 Sep 08 16:24:17.533696 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.533631 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerDied","Data":"62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612"} Sep 08 16:24:17.536528 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.536506 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bfdffcd8-k7frl" event={"ID":"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7","Type":"ContainerStarted","Data":"e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533"} Sep 08 16:24:17.539911 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.539889 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-58zzz" event={"ID":"97526267-ea08-4c82-aa89-5aeee4646510","Type":"ContainerStarted","Data":"a67ec5cba9a1f2e3b3c40be2e462995b2ec92c6b152f58161a3253509eff0982"} Sep 08 16:24:17.540012 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.539932 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-58zzz" event={"ID":"97526267-ea08-4c82-aa89-5aeee4646510","Type":"ContainerStarted","Data":"c073599a94ce8fc78c4b56c18c81f3faf9d73dd3f818b7da0b5e2ea4e784383d"} Sep 08 16:24:17.544536 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.543971 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" event={"ID":"bfb80ebb-1763-4fa4-9d57-3bb46df66d79","Type":"ContainerStarted","Data":"24b6bfd3b508071643eff7ed05be713aa477a7686634b8b897637e3cca0091e4"} Sep 08 16:24:17.547619 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.547576 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gc886" podStartSLOduration=15.422141365 podStartE2EDuration="17.547545059s" totalImagesPullingTime="2.125403694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:17.54554636 +0000 UTC m=+85.016532853" watchObservedRunningTime="2026-09-08 16:24:17.547545059 +0000 UTC m=+85.018531516" Sep 08 16:24:17.555216 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.555194 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" event={"ID":"f517b143-d4bb-496d-9838-c554d4babd5b","Type":"ContainerStarted","Data":"0d4cbe7b39282af919d9203afadc611d4142a290a1e8868ad30a08d233ca5f14"} Sep 08 16:24:17.555440 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.555408 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" event={"ID":"f517b143-d4bb-496d-9838-c554d4babd5b","Type":"ContainerStarted","Data":"715b1ea40711d5377e282efba7982f0a73f52221c175dc6214e153b30f4d7a30"} Sep 08 16:24:17.555628 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.555595 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" event={"ID":"f517b143-d4bb-496d-9838-c554d4babd5b","Type":"ContainerStarted","Data":"48285b04f079bb8712d0da37451e14f383b36549134ab2e3c720a8a284264187"} Sep 08 16:24:17.561543 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.561519 2567 generic.go:356] "Generic (PLEG): container finished" podID="a68654cd-4b38-4118-87d9-537792259b79" containerID="4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa" exitCode=0 Sep 08 16:24:17.561683 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.561606 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerDied","Data":"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa"} Sep 08 16:24:17.561683 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.561668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerStarted","Data":"da248b00256c7aa759ac13c69de9b82f5852b65bbdd58ac6262f9ceccfa65348"} Sep 08 16:24:17.569485 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.569437 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xs9n4" podStartSLOduration=5.53459436 podStartE2EDuration="17.56941835s" totalImagesPullingTime="12.03482399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:17.568824405 +0000 UTC m=+85.039810875" watchObservedRunningTime="2026-09-08 16:24:17.56941835 +0000 UTC m=+85.040404808" Sep 08 16:24:17.570106 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.569900 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-thmg6" event={"ID":"2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed","Type":"ContainerStarted","Data":"15285fe12b83e0c6e594e0c686d1a0150c68f230e711c96014e6c913e54339bd"} Sep 08 16:24:17.570499 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.570478 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-thmg6" Sep 08 16:24:17.576360 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.575208 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-thmg6" Sep 08 16:24:17.590455 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.590404 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84bfdffcd8-k7frl" podStartSLOduration=3.6521647059999998 podStartE2EDuration="16.590384409s" totalImagesPullingTime="12.938219703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:17.589979673 +0000 UTC m=+85.060966127" watchObservedRunningTime="2026-09-08 16:24:17.590384409 +0000 UTC m=+85.061370867" Sep 08 16:24:17.611671 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.611415 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-58zzz" podStartSLOduration=15.458612557 podStartE2EDuration="17.611392215s" totalImagesPullingTime="2.044647402s" totalInitContainerRuntime="108.132256ms" isStatefulPod=true podCreationTimestamp="2026-09-08 16:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:17.609466912 +0000 UTC m=+85.080453369" watchObservedRunningTime="2026-09-08 16:24:17.611392215 +0000 UTC m=+85.082378671" Sep 08 16:24:17.655191 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.655124 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" podStartSLOduration=3.470825937 podStartE2EDuration="13.655102123s" totalImagesPullingTime="10.184276186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:17.654139956 +0000 UTC m=+85.125126412" watchObservedRunningTime="2026-09-08 16:24:17.655102123 +0000 UTC m=+85.126088579" Sep 08 16:24:17.705958 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:17.705853 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-thmg6" podStartSLOduration=2.616275797 podStartE2EDuration="27.705831198s" totalImagesPullingTime="25.089555401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:17.674322362 +0000 UTC m=+85.145308816" watchObservedRunningTime="2026-09-08 16:24:17.705831198 +0000 UTC m=+85.176817655" Sep 08 16:24:21.608269 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.601722 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" event={"ID":"913426d9-bf7f-443e-8da2-d68e6b953e1f","Type":"ContainerStarted","Data":"e50e081e668c5d3ef0b5e795917e1274fff45f645b34f00c90575901f1a068ba"} Sep 08 16:24:21.608269 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.605857 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" event={"ID":"f517b143-d4bb-496d-9838-c554d4babd5b","Type":"ContainerStarted","Data":"5844a30b0b93f1184cbff29ec243c79d7c03dd0ce7bc2fbc8c41e792f1fee8e0"} Sep 08 16:24:21.608269 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.605927 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" event={"ID":"f517b143-d4bb-496d-9838-c554d4babd5b","Type":"ContainerStarted","Data":"7edd2ba4fb57e4336a179dd5e40feefb0e5c484bbab2c5c86bc62c9904bd787a"} Sep 08 16:24:21.608989 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.608958 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" event={"ID":"614b8d36-dbe9-49fb-b0c0-5da6e73987ea","Type":"ContainerStarted","Data":"969abde1fc3a9e37b4de6d68324296b9ca5b0ed2923b119b0df8cbe422511333"} Sep 08 16:24:21.609983 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.609910 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:21.618001 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.617975 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" Sep 08 16:24:21.622273 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.622248 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:21.622384 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.622293 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:21.628228 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.628199 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:21.634044 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.633956 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sqqpm" podStartSLOduration=11.903240183 podStartE2EDuration="16.63394048s" totalImagesPullingTime="4.730700297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:21.631232084 +0000 UTC m=+89.102218541" watchObservedRunningTime="2026-09-08 16:24:21.63394048 +0000 UTC m=+89.104926937" Sep 08 16:24:21.637768 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:21.637681 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:22.569202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:22.569128 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84bfdffcd8-k7frl"] Sep 08 16:24:22.621103 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:22.618458 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" event={"ID":"f517b143-d4bb-496d-9838-c554d4babd5b","Type":"ContainerStarted","Data":"d1469d36645e90fb601822e8a59122216dec7865ec97c38e45dd3d3c29867e82"} Sep 08 16:24:22.621103 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:22.619798 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:22.623053 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:22.622954 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" event={"ID":"913426d9-bf7f-443e-8da2-d68e6b953e1f","Type":"ContainerStarted","Data":"8dcd55e5d56165244f0d5aa4d3f1258bba96a6784568cf6185b0e12c949576d0"} Sep 08 16:24:22.627475 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:22.626614 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerStarted","Data":"cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b"} Sep 08 16:24:22.627808 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:22.627786 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" Sep 08 16:24:22.677752 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:22.677688 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d88d5644f-x68ct" podStartSLOduration=3.644778198 podStartE2EDuration="19.677665068s" totalImagesPullingTime="16.03288687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:22.650696161 +0000 UTC m=+90.121682618" watchObservedRunningTime="2026-09-08 16:24:22.677665068 +0000 UTC m=+90.148651525" Sep 08 16:24:23.634630 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:23.634544 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" event={"ID":"913426d9-bf7f-443e-8da2-d68e6b953e1f","Type":"ContainerStarted","Data":"05227c43f765b797d9499c2c101849461dfb0347d2cc608291bf3de232a578ed"} Sep 08 16:24:23.638101 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:23.638067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerStarted","Data":"4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c"} Sep 08 16:24:23.638230 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:23.638162 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerStarted","Data":"80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff"} Sep 08 16:24:23.663791 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:23.663724 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d74d56774-hkgxr" podStartSLOduration=14.229719424 podStartE2EDuration="18.663700695s" totalImagesPullingTime="4.433981271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:23.662999236 +0000 UTC m=+91.133985693" watchObservedRunningTime="2026-09-08 16:24:23.663700695 +0000 UTC m=+91.134687153" Sep 08 16:24:24.649306 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.649260 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerStarted","Data":"e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008"} Sep 08 16:24:24.649747 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.649393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerStarted","Data":"2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33"} Sep 08 16:24:24.649747 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.649432 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerStarted","Data":"bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9"} Sep 08 16:24:24.653938 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.653902 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerStarted","Data":"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb"} Sep 08 16:24:24.654117 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.653985 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerStarted","Data":"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be"} Sep 08 16:24:24.654117 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.654054 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerStarted","Data":"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9"} Sep 08 16:24:24.654117 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.654092 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerStarted","Data":"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00"} Sep 08 16:24:24.654271 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.654127 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerStarted","Data":"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9"} Sep 08 16:24:24.654271 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.654169 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerStarted","Data":"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224"} Sep 08 16:24:24.678691 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.678628 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.358471428 podStartE2EDuration="23.678608807s" totalImagesPullingTime="16.294082831s" totalInitContainerRuntime="26.054548ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:24.67774702 +0000 UTC m=+92.148733478" watchObservedRunningTime="2026-09-08 16:24:24.678608807 +0000 UTC m=+92.149595265" Sep 08 16:24:24.710619 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:24.710551 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=12.574136991 podStartE2EDuration="18.710529476s" totalImagesPullingTime="6.104494633s" totalInitContainerRuntime="31.897852ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:24.70860801 +0000 UTC m=+92.179594503" watchObservedRunningTime="2026-09-08 16:24:24.710529476 +0000 UTC m=+92.181515933" Sep 08 16:24:25.305370 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:25.305324 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:25.305552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:25.305416 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:27.086513 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.086475 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-777b95444c-2zpfb"] Sep 08 16:24:27.123715 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.123690 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777b95444c-2zpfb"] Sep 08 16:24:27.123868 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.123776 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.131191 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.131170 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 16:24:27.142754 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.142733 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:24:27.293727 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.293698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-oauth-serving-cert\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.293964 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.293940 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-service-ca\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.294080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.294006 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-trusted-ca-bundle\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.294177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.294161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mphd\" (UniqueName: \"kubernetes.io/projected/bd8ba63f-841a-4789-a762-46fc1a3a004f-kube-api-access-6mphd\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.294329 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.294313 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-config\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.294404 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.294374 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-oauth-config\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.294741 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.294516 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-serving-cert\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.395634 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.395555 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mphd\" (UniqueName: \"kubernetes.io/projected/bd8ba63f-841a-4789-a762-46fc1a3a004f-kube-api-access-6mphd\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.395634 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.395616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-config\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.395838 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.395648 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-oauth-config\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.395838 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.395684 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-serving-cert\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.395838 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.395729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-oauth-serving-cert\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.395838 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.395792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-service-ca\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.395838 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.395815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-trusted-ca-bundle\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.396509 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.396482 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-service-ca\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.396613 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.396546 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-oauth-serving-cert\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.396613 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.396546 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-config\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.396918 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.396895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-trusted-ca-bundle\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.398400 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.398369 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-oauth-config\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.398493 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.398424 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-serving-cert\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.404101 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.404082 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mphd\" (UniqueName: \"kubernetes.io/projected/bd8ba63f-841a-4789-a762-46fc1a3a004f-kube-api-access-6mphd\") pod \"console-777b95444c-2zpfb\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.433143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.433116 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:27.741112 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:27.741080 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777b95444c-2zpfb"] Sep 08 16:24:27.744292 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:24:27.744254 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8ba63f_841a_4789_a762_46fc1a3a004f.slice/crio-cbebe935fcbe2fa7b9a5f1838b335fddaf17263eeea8f1648e51388e0117a541 WatchSource:0}: Error finding container cbebe935fcbe2fa7b9a5f1838b335fddaf17263eeea8f1648e51388e0117a541: Status 404 returned error can't find the container with id cbebe935fcbe2fa7b9a5f1838b335fddaf17263eeea8f1648e51388e0117a541 Sep 08 16:24:28.175983 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:28.175959 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1695e5ed-a7fd-412d-a954-650bbc051e90/init-config-reloader/0.log" Sep 08 16:24:28.375011 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:28.374981 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1695e5ed-a7fd-412d-a954-650bbc051e90/alertmanager/0.log" Sep 08 16:24:28.576050 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:28.575952 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1695e5ed-a7fd-412d-a954-650bbc051e90/config-reloader/0.log" Sep 08 16:24:28.674383 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:28.674340 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777b95444c-2zpfb" event={"ID":"bd8ba63f-841a-4789-a762-46fc1a3a004f","Type":"ContainerStarted","Data":"7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa"} Sep 08 16:24:28.674383 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:28.674388 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777b95444c-2zpfb" event={"ID":"bd8ba63f-841a-4789-a762-46fc1a3a004f","Type":"ContainerStarted","Data":"cbebe935fcbe2fa7b9a5f1838b335fddaf17263eeea8f1648e51388e0117a541"} Sep 08 16:24:28.696850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:28.696797 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-777b95444c-2zpfb" podStartSLOduration=1.6967822479999999 podStartE2EDuration="1.696782248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:28.695544359 +0000 UTC m=+96.166530816" watchObservedRunningTime="2026-09-08 16:24:28.696782248 +0000 UTC m=+96.167768703" Sep 08 16:24:28.775536 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:28.775504 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1695e5ed-a7fd-412d-a954-650bbc051e90/kube-rbac-proxy-web/0.log" Sep 08 16:24:28.975873 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:28.975845 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1695e5ed-a7fd-412d-a954-650bbc051e90/kube-rbac-proxy/0.log" Sep 08 16:24:29.176417 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:29.176390 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1695e5ed-a7fd-412d-a954-650bbc051e90/kube-rbac-proxy-metric/0.log" Sep 08 16:24:29.377327 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:29.377253 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1695e5ed-a7fd-412d-a954-650bbc051e90/prom-label-proxy/0.log" Sep 08 16:24:29.777617 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:29.777593 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gc886_c92f55be-0da5-4c92-81bf-93ae92a68386/kube-state-metrics/0.log" Sep 08 16:24:29.975847 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:29.975818 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gc886_c92f55be-0da5-4c92-81bf-93ae92a68386/kube-rbac-proxy-main/0.log" Sep 08 16:24:30.175731 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:30.175677 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gc886_c92f55be-0da5-4c92-81bf-93ae92a68386/kube-rbac-proxy-self/0.log" Sep 08 16:24:30.375321 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:30.375290 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fc5949489-w2wk8_a335b4f2-2d52-4f21-95ad-c4f76259c0db/metrics-server/0.log" Sep 08 16:24:30.574896 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:30.574829 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-sqqpm_614b8d36-dbe9-49fb-b0c0-5da6e73987ea/monitoring-plugin/0.log" Sep 08 16:24:30.775487 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:30.775456 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-58zzz_97526267-ea08-4c82-aa89-5aeee4646510/init-textfile/0.log" Sep 08 16:24:30.974978 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:30.974951 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-58zzz_97526267-ea08-4c82-aa89-5aeee4646510/node-exporter/0.log" Sep 08 16:24:31.175227 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:31.175204 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-58zzz_97526267-ea08-4c82-aa89-5aeee4646510/kube-rbac-proxy/0.log" Sep 08 16:24:32.576771 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:32.576736 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xs9n4_bfb80ebb-1763-4fa4-9d57-3bb46df66d79/kube-rbac-proxy-main/0.log" Sep 08 16:24:32.774219 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:32.774190 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xs9n4_bfb80ebb-1763-4fa4-9d57-3bb46df66d79/kube-rbac-proxy-self/0.log" Sep 08 16:24:32.974263 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:32.974236 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xs9n4_bfb80ebb-1763-4fa4-9d57-3bb46df66d79/openshift-state-metrics/0.log" Sep 08 16:24:33.174769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:33.174740 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a68654cd-4b38-4118-87d9-537792259b79/init-config-reloader/0.log" Sep 08 16:24:33.376205 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:33.376128 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a68654cd-4b38-4118-87d9-537792259b79/prometheus/0.log" Sep 08 16:24:33.574530 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:33.574502 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a68654cd-4b38-4118-87d9-537792259b79/config-reloader/0.log" Sep 08 16:24:33.775104 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:33.775077 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a68654cd-4b38-4118-87d9-537792259b79/thanos-sidecar/0.log" Sep 08 16:24:33.987337 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:33.987312 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a68654cd-4b38-4118-87d9-537792259b79/kube-rbac-proxy-web/0.log" Sep 08 16:24:34.175330 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:34.175300 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a68654cd-4b38-4118-87d9-537792259b79/kube-rbac-proxy/0.log" Sep 08 16:24:34.375280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:34.375248 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a68654cd-4b38-4118-87d9-537792259b79/kube-rbac-proxy-thanos/0.log" Sep 08 16:24:34.578796 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:34.578701 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wbhl9_f4fa655f-11ee-49c0-9f94-914abaf95664/prometheus-operator/0.log" Sep 08 16:24:34.776855 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:34.776813 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wbhl9_f4fa655f-11ee-49c0-9f94-914abaf95664/kube-rbac-proxy/0.log" Sep 08 16:24:35.174620 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:35.174590 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d74d56774-hkgxr_913426d9-bf7f-443e-8da2-d68e6b953e1f/telemeter-client/0.log" Sep 08 16:24:35.375126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:35.375098 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d74d56774-hkgxr_913426d9-bf7f-443e-8da2-d68e6b953e1f/reload/0.log" Sep 08 16:24:35.574697 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:35.574614 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d74d56774-hkgxr_913426d9-bf7f-443e-8da2-d68e6b953e1f/kube-rbac-proxy/0.log" Sep 08 16:24:35.774952 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:35.774920 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/thanos-query/0.log" Sep 08 16:24:35.975041 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:35.974987 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/kube-rbac-proxy-web/0.log" Sep 08 16:24:36.175131 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:36.175103 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/kube-rbac-proxy/0.log" Sep 08 16:24:36.374465 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:36.374385 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/prom-label-proxy/0.log" Sep 08 16:24:36.453279 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:36.453254 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hp42x" Sep 08 16:24:36.574995 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:36.574970 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/kube-rbac-proxy-rules/0.log" Sep 08 16:24:36.778523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:36.778489 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/kube-rbac-proxy-metrics/0.log" Sep 08 16:24:37.433424 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:37.433387 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:37.433898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:37.433462 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:37.438233 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:37.438208 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:37.716919 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:37.716845 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:24:37.775430 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:37.775404 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-777b95444c-2zpfb_bd8ba63f-841a-4789-a762-46fc1a3a004f/console/0.log" Sep 08 16:24:37.976432 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:37.976339 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84bfdffcd8-k7frl_5b41b504-d2a7-437b-ba03-f55ef7a2a9a7/console/0.log" Sep 08 16:24:38.176658 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:38.176626 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-thmg6_2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed/download-server/0.log" Sep 08 16:24:38.974874 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:38.974847 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zvhbh_99976398-38d1-4613-891d-cb52ffd9af35/serve-healthcheck-canary/0.log" Sep 08 16:24:45.311051 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:45.311003 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:45.314840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:45.314817 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7fc5949489-w2wk8" Sep 08 16:24:48.665648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.665555 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84bfdffcd8-k7frl" podUID="5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" containerName="console" containerID="cri-o://e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533" gracePeriod=15 Sep 08 16:24:48.938586 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.938521 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:48.994224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994195 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-oauth-serving-cert\") pod \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " Sep 08 16:24:48.994394 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994242 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-config\") pod \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " Sep 08 16:24:48.994394 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994262 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6cd4\" (UniqueName: \"kubernetes.io/projected/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-kube-api-access-s6cd4\") pod \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " Sep 08 16:24:48.994497 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994409 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-serving-cert\") pod \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " Sep 08 16:24:48.994497 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994480 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-service-ca\") pod \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " Sep 08 16:24:48.994605 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994514 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-oauth-config\") pod \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\" (UID: \"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7\") " Sep 08 16:24:48.994657 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994591 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" (UID: "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:24:48.994737 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994708 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-config" (OuterVolumeSpecName: "console-config") pod "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" (UID: "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:24:48.994845 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994830 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-oauth-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:24:48.994904 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994851 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:24:48.994904 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.994848 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" (UID: "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:24:48.996949 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.996924 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" (UID: "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:24:48.997106 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.997085 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-kube-api-access-s6cd4" (OuterVolumeSpecName: "kube-api-access-s6cd4") pod "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" (UID: "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7"). InnerVolumeSpecName "kube-api-access-s6cd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:24:48.997170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:48.997118 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" (UID: "5b41b504-d2a7-437b-ba03-f55ef7a2a9a7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:24:49.095985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.095954 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6cd4\" (UniqueName: \"kubernetes.io/projected/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-kube-api-access-s6cd4\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:24:49.095985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.095981 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:24:49.095985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.095991 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:24:49.096204 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.096000 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7-console-oauth-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:24:49.772850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.772797 2567 generic.go:356] "Generic (PLEG): container finished" podID="5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" containerID="e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533" exitCode=0 Sep 08 16:24:49.773309 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.772855 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bfdffcd8-k7frl" Sep 08 16:24:49.773309 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.772876 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bfdffcd8-k7frl" event={"ID":"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7","Type":"ContainerDied","Data":"e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533"} Sep 08 16:24:49.773309 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.772942 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bfdffcd8-k7frl" event={"ID":"5b41b504-d2a7-437b-ba03-f55ef7a2a9a7","Type":"ContainerDied","Data":"51b655c576a2e77881b3bd8bb39c027b629e8db6102542d40526dda5132ed89c"} Sep 08 16:24:49.773309 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.772981 2567 scope.go:117] "RemoveContainer" containerID="e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533" Sep 08 16:24:49.781626 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.781605 2567 scope.go:117] "RemoveContainer" containerID="e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533" Sep 08 16:24:49.781927 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:24:49.781908 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533\": container with ID starting with e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533 not found: ID does not exist" containerID="e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533" Sep 08 16:24:49.781979 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.781936 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533"} err="failed to get container status \"e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533\": rpc error: code = NotFound desc = could not find container \"e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533\": container with ID starting with e8aed53e801558e82f726d6bba977fb028dc5aee000cf77e20cd6d7d3512e533 not found: ID does not exist" Sep 08 16:24:49.796467 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.796446 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84bfdffcd8-k7frl"] Sep 08 16:24:49.806459 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:49.806438 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84bfdffcd8-k7frl"] Sep 08 16:24:51.052355 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:24:51.052326 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" path="/var/lib/kubelet/pods/5b41b504-d2a7-437b-ba03-f55ef7a2a9a7/volumes" Sep 08 16:25:07.143566 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:07.143529 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:07.220497 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:07.220467 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:07.868094 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:07.868060 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:20.913539 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:20.913502 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:25:20.914060 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:20.913823 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="alertmanager" containerID="cri-o://cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b" gracePeriod=120 Sep 08 16:25:20.914060 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:20.913823 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy" containerID="cri-o://bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9" gracePeriod=120 Sep 08 16:25:20.914060 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:20.913852 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="config-reloader" containerID="cri-o://80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff" gracePeriod=120 Sep 08 16:25:20.914060 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:20.913877 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy-web" containerID="cri-o://4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c" gracePeriod=120 Sep 08 16:25:20.914060 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:20.913896 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="prom-label-proxy" containerID="cri-o://e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008" gracePeriod=120 Sep 08 16:25:20.914060 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:20.913854 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy-metric" containerID="cri-o://2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33" gracePeriod=120 Sep 08 16:25:21.908461 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:21.908432 2567 generic.go:356] "Generic (PLEG): container finished" podID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerID="e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008" exitCode=0 Sep 08 16:25:21.908461 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:21.908456 2567 generic.go:356] "Generic (PLEG): container finished" podID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerID="bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9" exitCode=0 Sep 08 16:25:21.908461 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:21.908462 2567 generic.go:356] "Generic (PLEG): container finished" podID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerID="80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff" exitCode=0 Sep 08 16:25:21.908461 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:21.908468 2567 generic.go:356] "Generic (PLEG): container finished" podID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerID="cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b" exitCode=0 Sep 08 16:25:21.908802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:21.908500 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerDied","Data":"e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008"} Sep 08 16:25:21.908802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:21.908570 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerDied","Data":"bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9"} Sep 08 16:25:21.908802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:21.908598 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerDied","Data":"80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff"} Sep 08 16:25:21.908802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:21.908624 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerDied","Data":"cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b"} Sep 08 16:25:22.162107 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.162051 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:22.177165 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177134 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-cluster-tls-config\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177276 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177180 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-main-tls\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177276 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177208 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177276 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177245 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgl9p\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-kube-api-access-xgl9p\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177416 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177298 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-config-volume\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177416 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177330 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-trusted-ca-bundle\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177416 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177359 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-metrics-client-ca\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177416 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177391 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-main-db\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177612 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177426 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-web-config\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177612 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177463 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177612 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177502 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-config-out\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177612 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177565 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-tls-assets\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177612 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177609 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy\") pod \"1695e5ed-a7fd-412d-a954-650bbc051e90\" (UID: \"1695e5ed-a7fd-412d-a954-650bbc051e90\") " Sep 08 16:25:22.177867 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177737 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:25:22.177925 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177883 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-metrics-client-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.177978 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.177931 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:25:22.178556 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.178303 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:25:22.180854 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.180817 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:22.181011 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.180986 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:25:22.181372 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.181320 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-config-out" (OuterVolumeSpecName: "config-out") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:25:22.181453 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.181419 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:22.181526 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.181504 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:22.181583 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.181565 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-kube-api-access-xgl9p" (OuterVolumeSpecName: "kube-api-access-xgl9p") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "kube-api-access-xgl9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:25:22.181645 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.181576 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:22.182390 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.182368 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-config-volume" (OuterVolumeSpecName: "config-volume") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:22.185199 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.185177 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:22.191847 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.191827 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-web-config" (OuterVolumeSpecName: "web-config") pod "1695e5ed-a7fd-412d-a954-650bbc051e90" (UID: "1695e5ed-a7fd-412d-a954-650bbc051e90"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:22.278480 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278446 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgl9p\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-kube-api-access-xgl9p\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278480 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278477 2567 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278488 2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278507 2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-alertmanager-main-db\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278515 2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-web-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278524 2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278532 2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1695e5ed-a7fd-412d-a954-650bbc051e90-config-out\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278540 2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1695e5ed-a7fd-412d-a954-650bbc051e90-tls-assets\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278547 2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278556 2567 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-cluster-tls-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278564 2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-main-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.278661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.278572 2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1695e5ed-a7fd-412d-a954-650bbc051e90-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:22.915725 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.915689 2567 generic.go:356] "Generic (PLEG): container finished" podID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerID="2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33" exitCode=0 Sep 08 16:25:22.915725 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.915724 2567 generic.go:356] "Generic (PLEG): container finished" podID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerID="4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c" exitCode=0 Sep 08 16:25:22.915929 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.915771 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerDied","Data":"2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33"} Sep 08 16:25:22.915929 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.915792 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:22.915929 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.915868 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerDied","Data":"4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c"} Sep 08 16:25:22.915929 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.915897 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1695e5ed-a7fd-412d-a954-650bbc051e90","Type":"ContainerDied","Data":"385e686f6dd19a95f8626e21118d04a9563b54ca3dbd342b6ff2062d7e6a4de9"} Sep 08 16:25:22.915929 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.915927 2567 scope.go:117] "RemoveContainer" containerID="e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008" Sep 08 16:25:22.924548 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.924530 2567 scope.go:117] "RemoveContainer" containerID="2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33" Sep 08 16:25:22.932340 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.932151 2567 scope.go:117] "RemoveContainer" containerID="bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9" Sep 08 16:25:22.938992 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.938972 2567 scope.go:117] "RemoveContainer" containerID="4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c" Sep 08 16:25:22.943236 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.943214 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:25:22.946987 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.946957 2567 scope.go:117] "RemoveContainer" containerID="80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff" Sep 08 16:25:22.952184 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.952161 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:25:22.955098 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.955079 2567 scope.go:117] "RemoveContainer" containerID="cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b" Sep 08 16:25:22.962371 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.962353 2567 scope.go:117] "RemoveContainer" containerID="62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612" Sep 08 16:25:22.975729 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.975712 2567 scope.go:117] "RemoveContainer" containerID="e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008" Sep 08 16:25:22.976007 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:22.975983 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008\": container with ID starting with e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008 not found: ID does not exist" containerID="e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008" Sep 08 16:25:22.976067 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.976044 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008"} err="failed to get container status \"e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008\": rpc error: code = NotFound desc = could not find container \"e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008\": container with ID starting with e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008 not found: ID does not exist" Sep 08 16:25:22.976167 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.976067 2567 scope.go:117] "RemoveContainer" containerID="2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33" Sep 08 16:25:22.976408 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:22.976387 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33\": container with ID starting with 2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33 not found: ID does not exist" containerID="2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33" Sep 08 16:25:22.976451 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.976429 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33"} err="failed to get container status \"2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33\": rpc error: code = NotFound desc = could not find container \"2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33\": container with ID starting with 2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33 not found: ID does not exist" Sep 08 16:25:22.976451 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.976446 2567 scope.go:117] "RemoveContainer" containerID="bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9" Sep 08 16:25:22.976732 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:22.976709 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9\": container with ID starting with bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9 not found: ID does not exist" containerID="bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9" Sep 08 16:25:22.976833 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.976736 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9"} err="failed to get container status \"bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9\": rpc error: code = NotFound desc = could not find container \"bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9\": container with ID starting with bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9 not found: ID does not exist" Sep 08 16:25:22.976833 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.976751 2567 scope.go:117] "RemoveContainer" containerID="4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c" Sep 08 16:25:22.977041 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:22.977008 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c\": container with ID starting with 4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c not found: ID does not exist" containerID="4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c" Sep 08 16:25:22.977162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.977049 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c"} err="failed to get container status \"4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c\": rpc error: code = NotFound desc = could not find container \"4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c\": container with ID starting with 4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c not found: ID does not exist" Sep 08 16:25:22.977162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.977071 2567 scope.go:117] "RemoveContainer" containerID="80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff" Sep 08 16:25:22.977387 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:22.977368 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff\": container with ID starting with 80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff not found: ID does not exist" containerID="80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff" Sep 08 16:25:22.977472 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.977394 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff"} err="failed to get container status \"80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff\": rpc error: code = NotFound desc = could not find container \"80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff\": container with ID starting with 80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff not found: ID does not exist" Sep 08 16:25:22.977472 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.977413 2567 scope.go:117] "RemoveContainer" containerID="cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b" Sep 08 16:25:22.977712 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:22.977683 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b\": container with ID starting with cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b not found: ID does not exist" containerID="cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b" Sep 08 16:25:22.977712 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.977719 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b"} err="failed to get container status \"cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b\": rpc error: code = NotFound desc = could not find container \"cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b\": container with ID starting with cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b not found: ID does not exist" Sep 08 16:25:22.978217 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.977739 2567 scope.go:117] "RemoveContainer" containerID="62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612" Sep 08 16:25:22.978217 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:22.978094 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612\": container with ID starting with 62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612 not found: ID does not exist" containerID="62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612" Sep 08 16:25:22.978217 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.978141 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612"} err="failed to get container status \"62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612\": rpc error: code = NotFound desc = could not find container \"62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612\": container with ID starting with 62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612 not found: ID does not exist" Sep 08 16:25:22.978217 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.978156 2567 scope.go:117] "RemoveContainer" containerID="e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008" Sep 08 16:25:22.978611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.978511 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008"} err="failed to get container status \"e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008\": rpc error: code = NotFound desc = could not find container \"e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008\": container with ID starting with e990757d62eaa8f2c5483749cea72f14343b809499fc238391545cdd548a0008 not found: ID does not exist" Sep 08 16:25:22.978611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.978537 2567 scope.go:117] "RemoveContainer" containerID="2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33" Sep 08 16:25:22.978858 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.978818 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33"} err="failed to get container status \"2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33\": rpc error: code = NotFound desc = could not find container \"2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33\": container with ID starting with 2285aab16bf53f6808d7e8dc3b086aabe5fed419f3fe29b25340ff6a33bf1a33 not found: ID does not exist" Sep 08 16:25:22.978858 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.978841 2567 scope.go:117] "RemoveContainer" containerID="bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9" Sep 08 16:25:22.979150 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.979130 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9"} err="failed to get container status \"bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9\": rpc error: code = NotFound desc = could not find container \"bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9\": container with ID starting with bb8571dc6f9c81cea04ca146abaa5c1bb4a4d68b10583eae6ab9259a1a78cdb9 not found: ID does not exist" Sep 08 16:25:22.979150 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.979149 2567 scope.go:117] "RemoveContainer" containerID="4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c" Sep 08 16:25:22.979436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.979412 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c"} err="failed to get container status \"4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c\": rpc error: code = NotFound desc = could not find container \"4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c\": container with ID starting with 4537c617958eb0822b62550457b1fb9576b5fc026b1071d30607f99043e9eb2c not found: ID does not exist" Sep 08 16:25:22.979504 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.979439 2567 scope.go:117] "RemoveContainer" containerID="80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff" Sep 08 16:25:22.979705 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.979686 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff"} err="failed to get container status \"80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff\": rpc error: code = NotFound desc = could not find container \"80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff\": container with ID starting with 80de5c5907947ef9f2e204bf54ff4101d59b0d0ebd734ad257a133ad3b0fe7ff not found: ID does not exist" Sep 08 16:25:22.979747 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.979705 2567 scope.go:117] "RemoveContainer" containerID="cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b" Sep 08 16:25:22.979959 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.979929 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b"} err="failed to get container status \"cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b\": rpc error: code = NotFound desc = could not find container \"cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b\": container with ID starting with cb5b484e35c9ffdf72416f3219e72261bb805af756f077a956eabf7c3778426b not found: ID does not exist" Sep 08 16:25:22.979959 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.979955 2567 scope.go:117] "RemoveContainer" containerID="62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612" Sep 08 16:25:22.980297 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980280 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612"} err="failed to get container status \"62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612\": rpc error: code = NotFound desc = could not find container \"62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612\": container with ID starting with 62d30f71559f1a08c6247efd839d829a13e0b9c4b80830f3993c039cf62ee612 not found: ID does not exist" Sep 08 16:25:22.980471 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980456 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:25:22.980765 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980754 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" containerName="console" Sep 08 16:25:22.980798 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980767 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" containerName="console" Sep 08 16:25:22.980798 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980777 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="alertmanager" Sep 08 16:25:22.980798 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980782 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="alertmanager" Sep 08 16:25:22.980798 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980788 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="config-reloader" Sep 08 16:25:22.980798 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980794 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="config-reloader" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980803 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="prom-label-proxy" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980809 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="prom-label-proxy" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980817 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="init-config-reloader" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980822 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="init-config-reloader" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980836 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy-metric" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980842 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy-metric" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980850 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy-web" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980855 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy-web" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980862 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980866 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980912 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy-metric" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980919 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b41b504-d2a7-437b-ba03-f55ef7a2a9a7" containerName="console" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980924 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="alertmanager" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980929 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy-web" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980934 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="config-reloader" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980939 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="kube-rbac-proxy" Sep 08 16:25:22.980985 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.980945 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" containerName="prom-label-proxy" Sep 08 16:25:22.986192 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.986176 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:22.989215 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.989185 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 08 16:25:22.989215 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.989203 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 08 16:25:22.989370 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.989220 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 08 16:25:22.989370 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.989267 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 08 16:25:22.989552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.989533 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 08 16:25:22.989688 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.989674 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 08 16:25:22.989866 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.989848 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 08 16:25:22.989923 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.989905 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 08 16:25:22.989923 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.989856 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c7gwg\"" Sep 08 16:25:22.996213 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:22.996195 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 08 16:25:23.001098 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.001078 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:25:23.052290 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.052261 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1695e5ed-a7fd-412d-a954-650bbc051e90" path="/var/lib/kubelet/pods/1695e5ed-a7fd-412d-a954-650bbc051e90/volumes" Sep 08 16:25:23.085702 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.085671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-web-config\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.085825 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.085713 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.085825 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.085733 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.085825 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.085781 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-config-volume\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.085825 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.085801 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37ef7003-0710-4399-94e9-db4fe1adcffa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.085825 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.085821 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.086072 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.085845 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37ef7003-0710-4399-94e9-db4fe1adcffa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.086072 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.085905 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/37ef7003-0710-4399-94e9-db4fe1adcffa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.086072 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.085954 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37ef7003-0710-4399-94e9-db4fe1adcffa-config-out\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.086072 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.086005 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.086072 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.086059 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9cv\" (UniqueName: \"kubernetes.io/projected/37ef7003-0710-4399-94e9-db4fe1adcffa-kube-api-access-cp9cv\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.086365 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.086104 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37ef7003-0710-4399-94e9-db4fe1adcffa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.086365 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.086122 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.186819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.186722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-web-config\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.186819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.186781 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.186819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.186802 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.186819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.186829 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-config-volume\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187443 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.186849 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37ef7003-0710-4399-94e9-db4fe1adcffa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187443 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.186865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187443 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.186880 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37ef7003-0710-4399-94e9-db4fe1adcffa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187443 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.186894 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/37ef7003-0710-4399-94e9-db4fe1adcffa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187443 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.186918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37ef7003-0710-4399-94e9-db4fe1adcffa-config-out\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187443 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.187391 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/37ef7003-0710-4399-94e9-db4fe1adcffa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187925 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.187517 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187925 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.187620 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9cv\" (UniqueName: \"kubernetes.io/projected/37ef7003-0710-4399-94e9-db4fe1adcffa-kube-api-access-cp9cv\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187925 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.187764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37ef7003-0710-4399-94e9-db4fe1adcffa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.187925 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.187797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.188168 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.187952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37ef7003-0710-4399-94e9-db4fe1adcffa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.189934 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.189906 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37ef7003-0710-4399-94e9-db4fe1adcffa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.190188 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.190145 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37ef7003-0710-4399-94e9-db4fe1adcffa-config-out\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.190301 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.190248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-web-config\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.190363 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.190300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.190778 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.190751 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.190932 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.190911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37ef7003-0710-4399-94e9-db4fe1adcffa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.190996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.190971 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.191129 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.191110 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.191197 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.191178 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-config-volume\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.192489 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.192467 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/37ef7003-0710-4399-94e9-db4fe1adcffa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.196091 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.196073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9cv\" (UniqueName: \"kubernetes.io/projected/37ef7003-0710-4399-94e9-db4fe1adcffa-kube-api-access-cp9cv\") pod \"alertmanager-main-0\" (UID: \"37ef7003-0710-4399-94e9-db4fe1adcffa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.297221 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.297186 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:25:23.577361 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.577280 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:25:23.581389 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:25:23.581361 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ef7003_0710_4399_94e9_db4fe1adcffa.slice/crio-8d6ad581b9959505009ab09468d12d4bd66a31f18e56e56a884e457b1b21e4ac WatchSource:0}: Error finding container 8d6ad581b9959505009ab09468d12d4bd66a31f18e56e56a884e457b1b21e4ac: Status 404 returned error can't find the container with id 8d6ad581b9959505009ab09468d12d4bd66a31f18e56e56a884e457b1b21e4ac Sep 08 16:25:23.921826 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.921797 2567 generic.go:356] "Generic (PLEG): container finished" podID="37ef7003-0710-4399-94e9-db4fe1adcffa" containerID="9f988e196f7b7582f627ef4222be03c83e53fbdc87049ef1485043792c9f6276" exitCode=0 Sep 08 16:25:23.921980 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.921859 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37ef7003-0710-4399-94e9-db4fe1adcffa","Type":"ContainerDied","Data":"9f988e196f7b7582f627ef4222be03c83e53fbdc87049ef1485043792c9f6276"} Sep 08 16:25:23.921980 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:23.921909 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37ef7003-0710-4399-94e9-db4fe1adcffa","Type":"ContainerStarted","Data":"8d6ad581b9959505009ab09468d12d4bd66a31f18e56e56a884e457b1b21e4ac"} Sep 08 16:25:24.929185 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:24.929150 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37ef7003-0710-4399-94e9-db4fe1adcffa","Type":"ContainerStarted","Data":"77b814d86fd22405f99d23b6a62d154d7c7f61459369834a9609906af7fb8dd3"} Sep 08 16:25:24.929571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:24.929218 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37ef7003-0710-4399-94e9-db4fe1adcffa","Type":"ContainerStarted","Data":"2307c0d912e5f1a5fe8fd6fec2930f83c08d5945850fe768cbdc1ee93e95e1ce"} Sep 08 16:25:24.929571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:24.929244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37ef7003-0710-4399-94e9-db4fe1adcffa","Type":"ContainerStarted","Data":"d4853bf339cc99c1c5755de8cc976dcf0acc4a40c0ff4430009a7a113a717686"} Sep 08 16:25:24.929571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:24.929268 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37ef7003-0710-4399-94e9-db4fe1adcffa","Type":"ContainerStarted","Data":"c4fed207f50ac7a1fdc9a5cabc62f3aeecfb18f646d32db1a9e543615751a226"} Sep 08 16:25:24.929571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:24.929291 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37ef7003-0710-4399-94e9-db4fe1adcffa","Type":"ContainerStarted","Data":"f947c6283b20f16827cfd2d6265bd3c3f8438fa397f01f9e3037349ea2276ff8"} Sep 08 16:25:24.929571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:24.929313 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37ef7003-0710-4399-94e9-db4fe1adcffa","Type":"ContainerStarted","Data":"ff41b59ba0a3320f9b0895d165f623d196e1c3bd73c94027f3ee0c577f58d5cd"} Sep 08 16:25:24.958995 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:24.958944 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.94320336 podStartE2EDuration="2.958926365s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.723005ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:25:24.957625022 +0000 UTC m=+152.428611475" watchObservedRunningTime="2026-09-08 16:25:24.958926365 +0000 UTC m=+152.429912859" Sep 08 16:25:25.161059 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.160997 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:25:25.161342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.161314 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy" containerID="cri-o://2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" gracePeriod=600 Sep 08 16:25:25.161410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.161367 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" gracePeriod=600 Sep 08 16:25:25.161410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.161398 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="config-reloader" containerID="cri-o://e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" gracePeriod=600 Sep 08 16:25:25.161555 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.161312 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="prometheus" containerID="cri-o://83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" gracePeriod=600 Sep 08 16:25:25.161555 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.161408 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy-web" containerID="cri-o://5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" gracePeriod=600 Sep 08 16:25:25.161555 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.161367 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="thanos-sidecar" containerID="cri-o://bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" gracePeriod=600 Sep 08 16:25:25.420642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.420611 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:25.507333 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507250 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-metrics-client-certs\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507333 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507289 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507333 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507317 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-2\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507345 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-thanos-prometheus-http-client-file\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507432 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-db\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507481 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-1\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507543 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-serving-certs-ca-bundle\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507578 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-kubelet-serving-ca-bundle\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507615 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjqk9\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-kube-api-access-pjqk9\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507651 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-trusted-ca-bundle\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507652 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:25:25.507850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507675 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-metrics-client-ca\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507714 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-config-out\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507740 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-0\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507772 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-kube-rbac-proxy\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507801 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-web-config\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.507850 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507840 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-tls-assets\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.508321 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507866 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-grpc-tls\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.508321 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507911 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-tls\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.508321 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507932 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.508321 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507954 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:25:25.508321 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507977 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-config\") pod \"a68654cd-4b38-4118-87d9-537792259b79\" (UID: \"a68654cd-4b38-4118-87d9-537792259b79\") " Sep 08 16:25:25.508321 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.507955 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:25:25.508609 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.508351 2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.508609 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.508373 2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.508609 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.508390 2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.508751 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.508654 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:25:25.508751 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.508665 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:25:25.508970 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.508941 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:25:25.509261 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.509235 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:25:25.510924 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.510314 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:25.510924 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.510677 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:25.510924 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.510691 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-config" (OuterVolumeSpecName: "config") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:25.510924 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.510782 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:25.511235 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.511211 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-kube-api-access-pjqk9" (OuterVolumeSpecName: "kube-api-access-pjqk9") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "kube-api-access-pjqk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:25:25.511636 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.511608 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:25:25.512439 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.512412 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:25.512741 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.512692 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:25.512846 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.512791 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:25:25.513815 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.513795 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:25.513934 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.513863 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:25.514532 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.514506 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-config-out" (OuterVolumeSpecName: "config-out") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:25:25.524003 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.523972 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-web-config" (OuterVolumeSpecName: "web-config") pod "a68654cd-4b38-4118-87d9-537792259b79" (UID: "a68654cd-4b38-4118-87d9-537792259b79"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:25.609607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609561 2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609602 2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-metrics-client-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609614 2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609624 2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609635 2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-db\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609645 2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609653 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjqk9\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-kube-api-access-pjqk9\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609663 2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609671 2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-configmap-metrics-client-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609678 2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a68654cd-4b38-4118-87d9-537792259b79-config-out\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609686 2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a68654cd-4b38-4118-87d9-537792259b79-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609694 2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-kube-rbac-proxy\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609702 2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-web-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609710 2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a68654cd-4b38-4118-87d9-537792259b79-tls-assets\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609717 2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-grpc-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609724 2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.609841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.609732 2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a68654cd-4b38-4118-87d9-537792259b79-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:25:25.936478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936448 2567 generic.go:356] "Generic (PLEG): container finished" podID="a68654cd-4b38-4118-87d9-537792259b79" containerID="4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" exitCode=0 Sep 08 16:25:25.936478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936471 2567 generic.go:356] "Generic (PLEG): container finished" podID="a68654cd-4b38-4118-87d9-537792259b79" containerID="2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" exitCode=0 Sep 08 16:25:25.936478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936477 2567 generic.go:356] "Generic (PLEG): container finished" podID="a68654cd-4b38-4118-87d9-537792259b79" containerID="5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" exitCode=0 Sep 08 16:25:25.936478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936482 2567 generic.go:356] "Generic (PLEG): container finished" podID="a68654cd-4b38-4118-87d9-537792259b79" containerID="bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" exitCode=0 Sep 08 16:25:25.936478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936487 2567 generic.go:356] "Generic (PLEG): container finished" podID="a68654cd-4b38-4118-87d9-537792259b79" containerID="e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" exitCode=0 Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936492 2567 generic.go:356] "Generic (PLEG): container finished" podID="a68654cd-4b38-4118-87d9-537792259b79" containerID="83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" exitCode=0 Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936522 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936529 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerDied","Data":"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb"} Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936597 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerDied","Data":"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be"} Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerDied","Data":"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9"} Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936648 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerDied","Data":"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00"} Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936667 2567 scope.go:117] "RemoveContainer" containerID="4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936671 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerDied","Data":"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9"} Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936695 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerDied","Data":"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224"} Sep 08 16:25:25.937100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.936719 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a68654cd-4b38-4118-87d9-537792259b79","Type":"ContainerDied","Data":"da248b00256c7aa759ac13c69de9b82f5852b65bbdd58ac6262f9ceccfa65348"} Sep 08 16:25:25.944944 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.944925 2567 scope.go:117] "RemoveContainer" containerID="2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" Sep 08 16:25:25.952800 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.952782 2567 scope.go:117] "RemoveContainer" containerID="5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" Sep 08 16:25:25.959430 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.959412 2567 scope.go:117] "RemoveContainer" containerID="bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" Sep 08 16:25:25.963230 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.963206 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:25:25.966947 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.966928 2567 scope.go:117] "RemoveContainer" containerID="e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" Sep 08 16:25:25.968306 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.968285 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:25:25.981814 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.981798 2567 scope.go:117] "RemoveContainer" containerID="83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" Sep 08 16:25:25.989274 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.989259 2567 scope.go:117] "RemoveContainer" containerID="4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa" Sep 08 16:25:25.996238 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.996222 2567 scope.go:117] "RemoveContainer" containerID="4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" Sep 08 16:25:25.996512 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:25.996493 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": container with ID starting with 4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb not found: ID does not exist" containerID="4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" Sep 08 16:25:25.996582 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.996520 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb"} err="failed to get container status \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": rpc error: code = NotFound desc = could not find container \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": container with ID starting with 4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb not found: ID does not exist" Sep 08 16:25:25.996582 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.996540 2567 scope.go:117] "RemoveContainer" containerID="2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" Sep 08 16:25:25.997254 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:25.996784 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": container with ID starting with 2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be not found: ID does not exist" containerID="2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" Sep 08 16:25:25.997833 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.997805 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:25:25.998523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998507 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="config-reloader" Sep 08 16:25:25.998604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998525 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="config-reloader" Sep 08 16:25:25.998604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998536 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy" Sep 08 16:25:25.998604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998542 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy" Sep 08 16:25:25.998604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998556 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy-thanos" Sep 08 16:25:25.998604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998562 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy-thanos" Sep 08 16:25:25.998604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998585 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="thanos-sidecar" Sep 08 16:25:25.998604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998592 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="thanos-sidecar" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998611 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="prometheus" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998617 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="prometheus" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998630 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy-web" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998636 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy-web" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998643 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="init-config-reloader" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998654 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="init-config-reloader" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998756 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="prometheus" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998769 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy-web" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998776 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="thanos-sidecar" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998785 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="config-reloader" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998792 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy" Sep 08 16:25:25.998907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.998800 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68654cd-4b38-4118-87d9-537792259b79" containerName="kube-rbac-proxy-thanos" Sep 08 16:25:26.002896 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:25.997270 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be"} err="failed to get container status \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": rpc error: code = NotFound desc = could not find container \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": container with ID starting with 2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be not found: ID does not exist" Sep 08 16:25:26.002896 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.000548 2567 scope.go:117] "RemoveContainer" containerID="5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" Sep 08 16:25:26.003139 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:26.003110 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": container with ID starting with 5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9 not found: ID does not exist" containerID="5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" Sep 08 16:25:26.003235 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.003153 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9"} err="failed to get container status \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": rpc error: code = NotFound desc = could not find container \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": container with ID starting with 5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9 not found: ID does not exist" Sep 08 16:25:26.003235 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.003175 2567 scope.go:117] "RemoveContainer" containerID="bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" Sep 08 16:25:26.003955 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:26.003887 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": container with ID starting with bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00 not found: ID does not exist" containerID="bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" Sep 08 16:25:26.004070 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.003962 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00"} err="failed to get container status \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": rpc error: code = NotFound desc = could not find container \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": container with ID starting with bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00 not found: ID does not exist" Sep 08 16:25:26.004070 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.003981 2567 scope.go:117] "RemoveContainer" containerID="e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" Sep 08 16:25:26.004395 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:26.004369 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": container with ID starting with e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9 not found: ID does not exist" containerID="e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" Sep 08 16:25:26.004485 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.004400 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9"} err="failed to get container status \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": rpc error: code = NotFound desc = could not find container \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": container with ID starting with e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9 not found: ID does not exist" Sep 08 16:25:26.004485 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.004420 2567 scope.go:117] "RemoveContainer" containerID="83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" Sep 08 16:25:26.004698 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:26.004676 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": container with ID starting with 83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224 not found: ID does not exist" containerID="83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" Sep 08 16:25:26.004756 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.004704 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224"} err="failed to get container status \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": rpc error: code = NotFound desc = could not find container \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": container with ID starting with 83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224 not found: ID does not exist" Sep 08 16:25:26.004756 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.004723 2567 scope.go:117] "RemoveContainer" containerID="4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa" Sep 08 16:25:26.005009 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:25:26.004988 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": container with ID starting with 4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa not found: ID does not exist" containerID="4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa" Sep 08 16:25:26.005099 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.005013 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa"} err="failed to get container status \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": rpc error: code = NotFound desc = could not find container \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": container with ID starting with 4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa not found: ID does not exist" Sep 08 16:25:26.005099 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.005054 2567 scope.go:117] "RemoveContainer" containerID="4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" Sep 08 16:25:26.005331 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.005312 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb"} err="failed to get container status \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": rpc error: code = NotFound desc = could not find container \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": container with ID starting with 4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb not found: ID does not exist" Sep 08 16:25:26.005387 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.005333 2567 scope.go:117] "RemoveContainer" containerID="2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" Sep 08 16:25:26.005558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.005540 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be"} err="failed to get container status \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": rpc error: code = NotFound desc = could not find container \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": container with ID starting with 2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be not found: ID does not exist" Sep 08 16:25:26.005618 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.005557 2567 scope.go:117] "RemoveContainer" containerID="5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" Sep 08 16:25:26.005805 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.005789 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9"} err="failed to get container status \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": rpc error: code = NotFound desc = could not find container \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": container with ID starting with 5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9 not found: ID does not exist" Sep 08 16:25:26.005849 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.005806 2567 scope.go:117] "RemoveContainer" containerID="bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" Sep 08 16:25:26.006055 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006035 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00"} err="failed to get container status \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": rpc error: code = NotFound desc = could not find container \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": container with ID starting with bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00 not found: ID does not exist" Sep 08 16:25:26.006105 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006057 2567 scope.go:117] "RemoveContainer" containerID="e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" Sep 08 16:25:26.006266 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006249 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9"} err="failed to get container status \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": rpc error: code = NotFound desc = could not find container \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": container with ID starting with e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9 not found: ID does not exist" Sep 08 16:25:26.006321 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006267 2567 scope.go:117] "RemoveContainer" containerID="83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" Sep 08 16:25:26.006519 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006499 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224"} err="failed to get container status \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": rpc error: code = NotFound desc = could not find container \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": container with ID starting with 83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224 not found: ID does not exist" Sep 08 16:25:26.006585 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006522 2567 scope.go:117] "RemoveContainer" containerID="4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa" Sep 08 16:25:26.006753 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006725 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa"} err="failed to get container status \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": rpc error: code = NotFound desc = could not find container \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": container with ID starting with 4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa not found: ID does not exist" Sep 08 16:25:26.006798 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006753 2567 scope.go:117] "RemoveContainer" containerID="4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" Sep 08 16:25:26.006971 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006954 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb"} err="failed to get container status \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": rpc error: code = NotFound desc = could not find container \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": container with ID starting with 4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb not found: ID does not exist" Sep 08 16:25:26.007047 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.006972 2567 scope.go:117] "RemoveContainer" containerID="2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" Sep 08 16:25:26.007184 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.007168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.007236 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.007201 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be"} err="failed to get container status \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": rpc error: code = NotFound desc = could not find container \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": container with ID starting with 2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be not found: ID does not exist" Sep 08 16:25:26.007236 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.007219 2567 scope.go:117] "RemoveContainer" containerID="5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" Sep 08 16:25:26.007496 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.007475 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9"} err="failed to get container status \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": rpc error: code = NotFound desc = could not find container \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": container with ID starting with 5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9 not found: ID does not exist" Sep 08 16:25:26.007600 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.007498 2567 scope.go:117] "RemoveContainer" containerID="bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" Sep 08 16:25:26.007905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.007880 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00"} err="failed to get container status \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": rpc error: code = NotFound desc = could not find container \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": container with ID starting with bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00 not found: ID does not exist" Sep 08 16:25:26.007905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.007904 2567 scope.go:117] "RemoveContainer" containerID="e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" Sep 08 16:25:26.008236 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.008216 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9"} err="failed to get container status \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": rpc error: code = NotFound desc = could not find container \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": container with ID starting with e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9 not found: ID does not exist" Sep 08 16:25:26.008322 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.008238 2567 scope.go:117] "RemoveContainer" containerID="83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" Sep 08 16:25:26.008514 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.008495 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224"} err="failed to get container status \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": rpc error: code = NotFound desc = could not find container \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": container with ID starting with 83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224 not found: ID does not exist" Sep 08 16:25:26.008514 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.008514 2567 scope.go:117] "RemoveContainer" containerID="4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa" Sep 08 16:25:26.008777 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.008762 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa"} err="failed to get container status \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": rpc error: code = NotFound desc = could not find container \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": container with ID starting with 4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa not found: ID does not exist" Sep 08 16:25:26.008825 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.008777 2567 scope.go:117] "RemoveContainer" containerID="4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" Sep 08 16:25:26.009003 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.008984 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb"} err="failed to get container status \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": rpc error: code = NotFound desc = could not find container \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": container with ID starting with 4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb not found: ID does not exist" Sep 08 16:25:26.009079 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009004 2567 scope.go:117] "RemoveContainer" containerID="2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" Sep 08 16:25:26.009311 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009285 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be"} err="failed to get container status \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": rpc error: code = NotFound desc = could not find container \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": container with ID starting with 2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be not found: ID does not exist" Sep 08 16:25:26.009402 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009313 2567 scope.go:117] "RemoveContainer" containerID="5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" Sep 08 16:25:26.009613 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009581 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9"} err="failed to get container status \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": rpc error: code = NotFound desc = could not find container \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": container with ID starting with 5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9 not found: ID does not exist" Sep 08 16:25:26.009723 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009621 2567 scope.go:117] "RemoveContainer" containerID="bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" Sep 08 16:25:26.009723 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009595 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 08 16:25:26.009852 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009781 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ww6bp\"" Sep 08 16:25:26.009912 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009904 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 08 16:25:26.009982 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009959 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 08 16:25:26.010060 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.009981 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 08 16:25:26.010122 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010081 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ecml97ua1ifr7\"" Sep 08 16:25:26.010122 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010097 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 08 16:25:26.010231 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010146 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00"} err="failed to get container status \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": rpc error: code = NotFound desc = could not find container \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": container with ID starting with bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00 not found: ID does not exist" Sep 08 16:25:26.010231 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010171 2567 scope.go:117] "RemoveContainer" containerID="e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" Sep 08 16:25:26.010231 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010190 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 08 16:25:26.010411 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010315 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 08 16:25:26.010411 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010371 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 08 16:25:26.010411 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010385 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 08 16:25:26.010559 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010478 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 08 16:25:26.010559 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010479 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9"} err="failed to get container status \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": rpc error: code = NotFound desc = could not find container \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": container with ID starting with e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9 not found: ID does not exist" Sep 08 16:25:26.010559 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010498 2567 scope.go:117] "RemoveContainer" containerID="83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" Sep 08 16:25:26.010746 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010591 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 08 16:25:26.010746 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010705 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 08 16:25:26.010859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010825 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224"} err="failed to get container status \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": rpc error: code = NotFound desc = could not find container \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": container with ID starting with 83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224 not found: ID does not exist" Sep 08 16:25:26.010859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.010853 2567 scope.go:117] "RemoveContainer" containerID="4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa" Sep 08 16:25:26.011224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.011199 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa"} err="failed to get container status \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": rpc error: code = NotFound desc = could not find container \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": container with ID starting with 4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa not found: ID does not exist" Sep 08 16:25:26.011224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.011225 2567 scope.go:117] "RemoveContainer" containerID="4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" Sep 08 16:25:26.011551 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.011526 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb"} err="failed to get container status \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": rpc error: code = NotFound desc = could not find container \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": container with ID starting with 4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb not found: ID does not exist" Sep 08 16:25:26.011551 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.011550 2567 scope.go:117] "RemoveContainer" containerID="2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" Sep 08 16:25:26.011872 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.011849 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be"} err="failed to get container status \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": rpc error: code = NotFound desc = could not find container \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": container with ID starting with 2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be not found: ID does not exist" Sep 08 16:25:26.011872 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.011869 2567 scope.go:117] "RemoveContainer" containerID="5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" Sep 08 16:25:26.012189 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.012169 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9"} err="failed to get container status \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": rpc error: code = NotFound desc = could not find container \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": container with ID starting with 5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9 not found: ID does not exist" Sep 08 16:25:26.012283 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.012190 2567 scope.go:117] "RemoveContainer" containerID="bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" Sep 08 16:25:26.012506 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.012481 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00"} err="failed to get container status \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": rpc error: code = NotFound desc = could not find container \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": container with ID starting with bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00 not found: ID does not exist" Sep 08 16:25:26.012506 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.012507 2567 scope.go:117] "RemoveContainer" containerID="e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" Sep 08 16:25:26.012798 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.012772 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9"} err="failed to get container status \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": rpc error: code = NotFound desc = could not find container \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": container with ID starting with e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9 not found: ID does not exist" Sep 08 16:25:26.012900 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.012799 2567 scope.go:117] "RemoveContainer" containerID="83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" Sep 08 16:25:26.013200 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.013173 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224"} err="failed to get container status \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": rpc error: code = NotFound desc = could not find container \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": container with ID starting with 83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224 not found: ID does not exist" Sep 08 16:25:26.013200 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.013200 2567 scope.go:117] "RemoveContainer" containerID="4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa" Sep 08 16:25:26.013841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.013821 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa"} err="failed to get container status \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": rpc error: code = NotFound desc = could not find container \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": container with ID starting with 4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa not found: ID does not exist" Sep 08 16:25:26.013841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.013841 2567 scope.go:117] "RemoveContainer" containerID="4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb" Sep 08 16:25:26.014115 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.014096 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb"} err="failed to get container status \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": rpc error: code = NotFound desc = could not find container \"4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb\": container with ID starting with 4340aac29d525be3076f8a90e08260df95dda6eb45c461b541280d1da2c1d6cb not found: ID does not exist" Sep 08 16:25:26.014115 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.014115 2567 scope.go:117] "RemoveContainer" containerID="2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be" Sep 08 16:25:26.014417 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.014399 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 08 16:25:26.014504 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.014396 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be"} err="failed to get container status \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": rpc error: code = NotFound desc = could not find container \"2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be\": container with ID starting with 2c7dd45415aad3e884c5f59f3711f92a6323309e51c6bca838de21f7bfa404be not found: ID does not exist" Sep 08 16:25:26.014504 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.014441 2567 scope.go:117] "RemoveContainer" containerID="5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9" Sep 08 16:25:26.014712 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.014687 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9"} err="failed to get container status \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": rpc error: code = NotFound desc = could not find container \"5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9\": container with ID starting with 5ebf8e58b74fd2586ce2479d7080093a85360ffc3cf5fd286b32b473d7498df9 not found: ID does not exist" Sep 08 16:25:26.014712 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.014711 2567 scope.go:117] "RemoveContainer" containerID="bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00" Sep 08 16:25:26.014981 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.014963 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00"} err="failed to get container status \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": rpc error: code = NotFound desc = could not find container \"bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00\": container with ID starting with bf53d6653f0c1127f51ec3569c24248e160dd1847ed7886428a1dac3f63f7b00 not found: ID does not exist" Sep 08 16:25:26.015071 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.014996 2567 scope.go:117] "RemoveContainer" containerID="e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9" Sep 08 16:25:26.015467 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.015326 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9"} err="failed to get container status \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": rpc error: code = NotFound desc = could not find container \"e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9\": container with ID starting with e0ed1fb5ea542838aba12f7c7f06279a8efbe316d77f3593b0d3eff39ab2daa9 not found: ID does not exist" Sep 08 16:25:26.015467 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.015352 2567 scope.go:117] "RemoveContainer" containerID="83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224" Sep 08 16:25:26.015951 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.015910 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224"} err="failed to get container status \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": rpc error: code = NotFound desc = could not find container \"83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224\": container with ID starting with 83b21908c7dda58d862516326878769dadbd028c0064cf49122f3b73c134c224 not found: ID does not exist" Sep 08 16:25:26.015951 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.015933 2567 scope.go:117] "RemoveContainer" containerID="4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa" Sep 08 16:25:26.017839 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.017161 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa"} err="failed to get container status \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": rpc error: code = NotFound desc = could not find container \"4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa\": container with ID starting with 4efb181a40fc013d4b6290a44b587d9b1221aebc307805609a8040c4ab50aaaa not found: ID does not exist" Sep 08 16:25:26.017839 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.017371 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 08 16:25:26.018702 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.018686 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:25:26.114165 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114128 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114165 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114372 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114183 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m888x\" (UniqueName: \"kubernetes.io/projected/117aaedb-2d68-4522-8526-e4c19471f9d1-kube-api-access-m888x\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114372 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114203 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114372 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114266 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-web-config\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114372 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114281 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114372 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114301 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114372 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114349 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-config\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114574 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114385 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114574 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114405 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/117aaedb-2d68-4522-8526-e4c19471f9d1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114574 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114420 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114574 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114574 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114501 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114574 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114735 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114595 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114735 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114613 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114735 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114631 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/117aaedb-2d68-4522-8526-e4c19471f9d1-config-out\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114735 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114735 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114693 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.114735 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.114713 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215529 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215459 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/117aaedb-2d68-4522-8526-e4c19471f9d1-config-out\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215529 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215496 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215529 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215518 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215684 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215627 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215684 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215677 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215765 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215765 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215719 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m888x\" (UniqueName: \"kubernetes.io/projected/117aaedb-2d68-4522-8526-e4c19471f9d1-kube-api-access-m888x\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215858 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215858 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215805 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-web-config\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.215858 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215830 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216000 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215862 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216000 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215900 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-config\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216000 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215930 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216000 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215963 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/117aaedb-2d68-4522-8526-e4c19471f9d1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216000 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.215994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216287 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.216083 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216287 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.216115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216287 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.216154 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216287 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.216183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216287 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.216206 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216287 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.216208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.216581 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.216412 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.218155 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.216642 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.218155 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.217548 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.218867 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.218643 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/117aaedb-2d68-4522-8526-e4c19471f9d1-config-out\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.218867 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.218782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.218867 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.218831 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.219926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.219088 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-web-config\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.219926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.219236 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.219926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.219349 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-config\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.219926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.219536 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/117aaedb-2d68-4522-8526-e4c19471f9d1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.219926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.219592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.219926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.219758 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.219926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.219812 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.221337 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.221314 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.221610 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.221590 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.221695 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.221679 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.222237 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.222222 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/117aaedb-2d68-4522-8526-e4c19471f9d1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.222773 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.222756 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/117aaedb-2d68-4522-8526-e4c19471f9d1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.228543 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.228518 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m888x\" (UniqueName: \"kubernetes.io/projected/117aaedb-2d68-4522-8526-e4c19471f9d1-kube-api-access-m888x\") pod \"prometheus-k8s-0\" (UID: \"117aaedb-2d68-4522-8526-e4c19471f9d1\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.320305 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.320268 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:25:26.608501 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.608475 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:25:26.610569 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:25:26.610540 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117aaedb_2d68_4522_8526_e4c19471f9d1.slice/crio-3b33753969f50374b72adddaec223ac5a0ffacd9fd02a8ec5aadce7d26d86e91 WatchSource:0}: Error finding container 3b33753969f50374b72adddaec223ac5a0ffacd9fd02a8ec5aadce7d26d86e91: Status 404 returned error can't find the container with id 3b33753969f50374b72adddaec223ac5a0ffacd9fd02a8ec5aadce7d26d86e91 Sep 08 16:25:26.941982 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.941948 2567 generic.go:356] "Generic (PLEG): container finished" podID="117aaedb-2d68-4522-8526-e4c19471f9d1" containerID="6b315d750c66dd254146c5b42603ce9cf6c30c68690586e0383bfc8e75567ac2" exitCode=0 Sep 08 16:25:26.942464 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.942044 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"117aaedb-2d68-4522-8526-e4c19471f9d1","Type":"ContainerDied","Data":"6b315d750c66dd254146c5b42603ce9cf6c30c68690586e0383bfc8e75567ac2"} Sep 08 16:25:26.942464 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:26.942105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"117aaedb-2d68-4522-8526-e4c19471f9d1","Type":"ContainerStarted","Data":"3b33753969f50374b72adddaec223ac5a0ffacd9fd02a8ec5aadce7d26d86e91"} Sep 08 16:25:27.053866 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:27.053838 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68654cd-4b38-4118-87d9-537792259b79" path="/var/lib/kubelet/pods/a68654cd-4b38-4118-87d9-537792259b79/volumes" Sep 08 16:25:27.950276 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:27.950232 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"117aaedb-2d68-4522-8526-e4c19471f9d1","Type":"ContainerStarted","Data":"87ea9d1bb96cc695421f0338f603bd0a4256594ce34532b4e65aada08fabf9f2"} Sep 08 16:25:27.950643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:27.950353 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"117aaedb-2d68-4522-8526-e4c19471f9d1","Type":"ContainerStarted","Data":"32e0cd281fae12e9b51e2dffe075ad6ccf405409890e1f853eeafabd39a1f5c8"} Sep 08 16:25:27.950643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:27.950398 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"117aaedb-2d68-4522-8526-e4c19471f9d1","Type":"ContainerStarted","Data":"c67e2e8974672ed749de0e6651c056fc5585bb225da6428f5fbd16ba0b338f17"} Sep 08 16:25:27.950643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:27.950435 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"117aaedb-2d68-4522-8526-e4c19471f9d1","Type":"ContainerStarted","Data":"a9f38633d91401edce2472025026da3bf3a9f9c0ceefc158bf2a2893a14d71db"} Sep 08 16:25:27.950643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:27.950469 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"117aaedb-2d68-4522-8526-e4c19471f9d1","Type":"ContainerStarted","Data":"f4d439d7d0ba9cb7b491a004aca56bff449a6fbdfd2e76a85d4c8dca87ebaaa6"} Sep 08 16:25:27.950643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:27.950503 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"117aaedb-2d68-4522-8526-e4c19471f9d1","Type":"ContainerStarted","Data":"63177794920d727dd5aac91a8d8bbd7fdabe5aa30cb5d81e797352793bfa5e84"} Sep 08 16:25:27.984075 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:27.984000 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.96203664 podStartE2EDuration="2.983983734s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.947094ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:25:27.982274598 +0000 UTC m=+155.453261099" watchObservedRunningTime="2026-09-08 16:25:27.983983734 +0000 UTC m=+155.454970217" Sep 08 16:25:31.321008 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:25:31.320970 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:26:01.899080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:01.899044 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777b95444c-2zpfb"] Sep 08 16:26:26.321013 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:26.320975 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:26:26.337262 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:26.337236 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:26:26.923777 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:26.923732 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-777b95444c-2zpfb" podUID="bd8ba63f-841a-4789-a762-46fc1a3a004f" containerName="console" containerID="cri-o://7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa" gracePeriod=15 Sep 08 16:26:27.163314 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.163279 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:26:27.195513 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.195448 2567 generic.go:356] "Generic (PLEG): container finished" podID="bd8ba63f-841a-4789-a762-46fc1a3a004f" containerID="7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa" exitCode=0 Sep 08 16:26:27.195513 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.195493 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777b95444c-2zpfb" Sep 08 16:26:27.195716 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.195543 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777b95444c-2zpfb" event={"ID":"bd8ba63f-841a-4789-a762-46fc1a3a004f","Type":"ContainerDied","Data":"7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa"} Sep 08 16:26:27.195716 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.195599 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777b95444c-2zpfb" event={"ID":"bd8ba63f-841a-4789-a762-46fc1a3a004f","Type":"ContainerDied","Data":"cbebe935fcbe2fa7b9a5f1838b335fddaf17263eeea8f1648e51388e0117a541"} Sep 08 16:26:27.195716 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.195626 2567 scope.go:117] "RemoveContainer" containerID="7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa" Sep 08 16:26:27.204391 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.204372 2567 scope.go:117] "RemoveContainer" containerID="7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa" Sep 08 16:26:27.204673 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:26:27.204653 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa\": container with ID starting with 7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa not found: ID does not exist" containerID="7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa" Sep 08 16:26:27.204745 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.204681 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa"} err="failed to get container status \"7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa\": rpc error: code = NotFound desc = could not find container \"7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa\": container with ID starting with 7656bb43db93b614a6ea340e5f3d921604de91ea05499c048910457b592559aa not found: ID does not exist" Sep 08 16:26:27.211541 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.211522 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:26:27.243410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243385 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-trusted-ca-bundle\") pod \"bd8ba63f-841a-4789-a762-46fc1a3a004f\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " Sep 08 16:26:27.243535 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243432 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-serving-cert\") pod \"bd8ba63f-841a-4789-a762-46fc1a3a004f\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " Sep 08 16:26:27.243535 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243456 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-config\") pod \"bd8ba63f-841a-4789-a762-46fc1a3a004f\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " Sep 08 16:26:27.243535 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243476 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-oauth-serving-cert\") pod \"bd8ba63f-841a-4789-a762-46fc1a3a004f\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " Sep 08 16:26:27.243535 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243506 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-service-ca\") pod \"bd8ba63f-841a-4789-a762-46fc1a3a004f\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " Sep 08 16:26:27.243749 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243573 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-oauth-config\") pod \"bd8ba63f-841a-4789-a762-46fc1a3a004f\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " Sep 08 16:26:27.243749 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243624 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mphd\" (UniqueName: \"kubernetes.io/projected/bd8ba63f-841a-4789-a762-46fc1a3a004f-kube-api-access-6mphd\") pod \"bd8ba63f-841a-4789-a762-46fc1a3a004f\" (UID: \"bd8ba63f-841a-4789-a762-46fc1a3a004f\") " Sep 08 16:26:27.243869 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243844 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bd8ba63f-841a-4789-a762-46fc1a3a004f" (UID: "bd8ba63f-841a-4789-a762-46fc1a3a004f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:26:27.243928 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243879 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-config" (OuterVolumeSpecName: "console-config") pod "bd8ba63f-841a-4789-a762-46fc1a3a004f" (UID: "bd8ba63f-841a-4789-a762-46fc1a3a004f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:26:27.243982 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.243938 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bd8ba63f-841a-4789-a762-46fc1a3a004f" (UID: "bd8ba63f-841a-4789-a762-46fc1a3a004f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:26:27.244585 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.244562 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-service-ca" (OuterVolumeSpecName: "service-ca") pod "bd8ba63f-841a-4789-a762-46fc1a3a004f" (UID: "bd8ba63f-841a-4789-a762-46fc1a3a004f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:26:27.244682 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.244654 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:26:27.244682 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.244674 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:26:27.244789 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.244688 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-oauth-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:26:27.244789 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.244700 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ba63f-841a-4789-a762-46fc1a3a004f-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:26:27.246349 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.246320 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bd8ba63f-841a-4789-a762-46fc1a3a004f" (UID: "bd8ba63f-841a-4789-a762-46fc1a3a004f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:26:27.246662 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.246642 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bd8ba63f-841a-4789-a762-46fc1a3a004f" (UID: "bd8ba63f-841a-4789-a762-46fc1a3a004f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:26:27.246732 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.246720 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8ba63f-841a-4789-a762-46fc1a3a004f-kube-api-access-6mphd" (OuterVolumeSpecName: "kube-api-access-6mphd") pod "bd8ba63f-841a-4789-a762-46fc1a3a004f" (UID: "bd8ba63f-841a-4789-a762-46fc1a3a004f"). InnerVolumeSpecName "kube-api-access-6mphd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:26:27.345466 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.345426 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-oauth-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:26:27.345466 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.345455 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mphd\" (UniqueName: \"kubernetes.io/projected/bd8ba63f-841a-4789-a762-46fc1a3a004f-kube-api-access-6mphd\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:26:27.345466 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.345466 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd8ba63f-841a-4789-a762-46fc1a3a004f-console-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:26:27.520334 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.520305 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777b95444c-2zpfb"] Sep 08 16:26:27.527724 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:27.527702 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-777b95444c-2zpfb"] Sep 08 16:26:29.051406 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:29.051375 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8ba63f-841a-4789-a762-46fc1a3a004f" path="/var/lib/kubelet/pods/bd8ba63f-841a-4789-a762-46fc1a3a004f/volumes" Sep 08 16:26:42.261560 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.261525 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4668l"] Sep 08 16:26:42.261991 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.261857 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8ba63f-841a-4789-a762-46fc1a3a004f" containerName="console" Sep 08 16:26:42.261991 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.261868 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8ba63f-841a-4789-a762-46fc1a3a004f" containerName="console" Sep 08 16:26:42.261991 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.261929 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8ba63f-841a-4789-a762-46fc1a3a004f" containerName="console" Sep 08 16:26:42.266432 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.266410 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" Sep 08 16:26:42.271363 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.271339 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-txs7c\"" Sep 08 16:26:42.271611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.271595 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 16:26:42.272278 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.272260 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 16:26:42.278693 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.278673 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4668l"] Sep 08 16:26:42.376914 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.376878 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-str7t\" (UniqueName: \"kubernetes.io/projected/896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3-kube-api-access-str7t\") pod \"cert-manager-cainjector-559cdd8c87-4668l\" (UID: \"896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" Sep 08 16:26:42.377095 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.376935 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4668l\" (UID: \"896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" Sep 08 16:26:42.478427 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.478387 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-str7t\" (UniqueName: \"kubernetes.io/projected/896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3-kube-api-access-str7t\") pod \"cert-manager-cainjector-559cdd8c87-4668l\" (UID: \"896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" Sep 08 16:26:42.478606 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.478461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4668l\" (UID: \"896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" Sep 08 16:26:42.488549 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.488514 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4668l\" (UID: \"896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" Sep 08 16:26:42.489648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.489628 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-str7t\" (UniqueName: \"kubernetes.io/projected/896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3-kube-api-access-str7t\") pod \"cert-manager-cainjector-559cdd8c87-4668l\" (UID: \"896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" Sep 08 16:26:42.585437 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.585369 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" Sep 08 16:26:42.864806 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:42.864777 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4668l"] Sep 08 16:26:42.867000 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:26:42.866969 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod896ec7b7_f82e_486f_b6b3_e2a6d4afe2a3.slice/crio-34431bc1e1a0ea69990909c23a98628e1ad75f6bbd0ef0d1348182c45a8921af WatchSource:0}: Error finding container 34431bc1e1a0ea69990909c23a98628e1ad75f6bbd0ef0d1348182c45a8921af: Status 404 returned error can't find the container with id 34431bc1e1a0ea69990909c23a98628e1ad75f6bbd0ef0d1348182c45a8921af Sep 08 16:26:43.273105 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:43.273061 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" event={"ID":"896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3","Type":"ContainerStarted","Data":"34431bc1e1a0ea69990909c23a98628e1ad75f6bbd0ef0d1348182c45a8921af"} Sep 08 16:26:47.295912 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.295877 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" event={"ID":"896ec7b7-f82e-486f-b6b3-e2a6d4afe2a3","Type":"ContainerStarted","Data":"1348094b461cb549ef2be158916fb0f2767d3869ed52370a005a597335039abf"} Sep 08 16:26:47.338457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.338397 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4668l" podStartSLOduration=1.107903644 podStartE2EDuration="5.338374955s" totalImagesPullingTime="4.230471311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:26:47.335473527 +0000 UTC m=+234.806459983" watchObservedRunningTime="2026-09-08 16:26:47.338374955 +0000 UTC m=+234.809361411" Sep 08 16:26:47.618346 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.618304 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2m2hz"] Sep 08 16:26:47.621813 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.621788 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2m2hz" Sep 08 16:26:47.624758 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.624737 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n2xbk\"" Sep 08 16:26:47.725989 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.725953 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6c9\" (UniqueName: \"kubernetes.io/projected/765473e5-f8d2-4ece-893a-fc9c875b362b-kube-api-access-gh6c9\") pod \"cert-manager-55b4d954d5-2m2hz\" (UID: \"765473e5-f8d2-4ece-893a-fc9c875b362b\") " pod="cert-manager/cert-manager-55b4d954d5-2m2hz" Sep 08 16:26:47.726181 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.725996 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/765473e5-f8d2-4ece-893a-fc9c875b362b-bound-sa-token\") pod \"cert-manager-55b4d954d5-2m2hz\" (UID: \"765473e5-f8d2-4ece-893a-fc9c875b362b\") " pod="cert-manager/cert-manager-55b4d954d5-2m2hz" Sep 08 16:26:47.743555 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.743525 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2m2hz"] Sep 08 16:26:47.827548 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.827515 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6c9\" (UniqueName: \"kubernetes.io/projected/765473e5-f8d2-4ece-893a-fc9c875b362b-kube-api-access-gh6c9\") pod \"cert-manager-55b4d954d5-2m2hz\" (UID: \"765473e5-f8d2-4ece-893a-fc9c875b362b\") " pod="cert-manager/cert-manager-55b4d954d5-2m2hz" Sep 08 16:26:47.827548 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.827551 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/765473e5-f8d2-4ece-893a-fc9c875b362b-bound-sa-token\") pod \"cert-manager-55b4d954d5-2m2hz\" (UID: \"765473e5-f8d2-4ece-893a-fc9c875b362b\") " pod="cert-manager/cert-manager-55b4d954d5-2m2hz" Sep 08 16:26:47.840032 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.839986 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/765473e5-f8d2-4ece-893a-fc9c875b362b-bound-sa-token\") pod \"cert-manager-55b4d954d5-2m2hz\" (UID: \"765473e5-f8d2-4ece-893a-fc9c875b362b\") " pod="cert-manager/cert-manager-55b4d954d5-2m2hz" Sep 08 16:26:47.840142 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.840119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6c9\" (UniqueName: \"kubernetes.io/projected/765473e5-f8d2-4ece-893a-fc9c875b362b-kube-api-access-gh6c9\") pod \"cert-manager-55b4d954d5-2m2hz\" (UID: \"765473e5-f8d2-4ece-893a-fc9c875b362b\") " pod="cert-manager/cert-manager-55b4d954d5-2m2hz" Sep 08 16:26:47.936438 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:47.936401 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2m2hz" Sep 08 16:26:48.222298 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:48.222202 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2m2hz"] Sep 08 16:26:48.225158 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:26:48.225130 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod765473e5_f8d2_4ece_893a_fc9c875b362b.slice/crio-16aced25603de65d6a8363c44cc6ed37e3cf9de85e78ff6d4684fed826b245ab WatchSource:0}: Error finding container 16aced25603de65d6a8363c44cc6ed37e3cf9de85e78ff6d4684fed826b245ab: Status 404 returned error can't find the container with id 16aced25603de65d6a8363c44cc6ed37e3cf9de85e78ff6d4684fed826b245ab Sep 08 16:26:48.302477 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:48.302444 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2m2hz" event={"ID":"765473e5-f8d2-4ece-893a-fc9c875b362b","Type":"ContainerStarted","Data":"5db3bacd0bb89d8f01735ebc0a06b2909485e81799f51e9130e2e44aa263a93b"} Sep 08 16:26:48.302827 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:48.302493 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2m2hz" event={"ID":"765473e5-f8d2-4ece-893a-fc9c875b362b","Type":"ContainerStarted","Data":"16aced25603de65d6a8363c44cc6ed37e3cf9de85e78ff6d4684fed826b245ab"} Sep 08 16:26:48.327769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:26:48.327414 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-2m2hz" podStartSLOduration=1.327390552 podStartE2EDuration="1.327390552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:26:48.32418885 +0000 UTC m=+235.795175309" watchObservedRunningTime="2026-09-08 16:26:48.327390552 +0000 UTC m=+235.798377016" Sep 08 16:27:19.202671 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.202593 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j"] Sep 08 16:27:19.206047 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.206011 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.223856 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.223825 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:27:19.228175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.228151 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 08 16:27:19.228330 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.228311 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 08 16:27:19.228551 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.228537 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g8gnn\"" Sep 08 16:27:19.239554 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.239535 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 08 16:27:19.239655 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.239634 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 08 16:27:19.249338 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.249317 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j"] Sep 08 16:27:19.250625 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.250592 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg"] Sep 08 16:27:19.254138 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.254119 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.282289 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:27:19.282255 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret" Sep 08 16:27:19.291582 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.291558 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bf0430c-4800-48e9-925b-55bf8fae9ff8-manifests\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.291643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.291591 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-cert\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.291643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.291608 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.291643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.291622 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz6tn\" (UniqueName: \"kubernetes.io/projected/6bf0430c-4800-48e9-925b-55bf8fae9ff8-kube-api-access-jz6tn\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.291748 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.291686 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-manager-config\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.291748 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.291708 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.291748 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.291746 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7km9z\" (UniqueName: \"kubernetes.io/projected/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-kube-api-access-7km9z\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.291840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.291765 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-metrics-cert\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.291840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.291786 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bf0430c-4800-48e9-925b-55bf8fae9ff8-tmp\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.306451 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:27:19.306416 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 08 16:27:19.306567 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:27:19.306465 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 08 16:27:19.306567 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:27:19.306465 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-mtxq4\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mtxq4\"" type="*v1.Secret" Sep 08 16:27:19.341904 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:27:19.341867 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" Sep 08 16:27:19.392574 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.392539 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.392721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.392598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7km9z\" (UniqueName: \"kubernetes.io/projected/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-kube-api-access-7km9z\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.392721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.392621 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-metrics-cert\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.392721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.392642 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bf0430c-4800-48e9-925b-55bf8fae9ff8-tmp\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.392721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.392678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bf0430c-4800-48e9-925b-55bf8fae9ff8-manifests\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.392721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.392701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-cert\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.392978 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.392723 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.392978 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.392739 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz6tn\" (UniqueName: \"kubernetes.io/projected/6bf0430c-4800-48e9-925b-55bf8fae9ff8-kube-api-access-jz6tn\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.392978 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.392815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-manager-config\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.393203 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.393136 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bf0430c-4800-48e9-925b-55bf8fae9ff8-manifests\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.393559 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.393535 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-manager-config\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.395144 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.395125 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bf0430c-4800-48e9-925b-55bf8fae9ff8-tmp\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:19.395288 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.395271 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-cert\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.395351 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.395310 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-metrics-cert\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.487551 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.487463 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg"] Sep 08 16:27:19.650155 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.650120 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7km9z\" (UniqueName: \"kubernetes.io/projected/8a13e4b9-6153-46fc-adb3-ff09dbe153d1-kube-api-access-7km9z\") pod \"lws-controller-manager-5ddb9f5cb8-lkz9j\" (UID: \"8a13e4b9-6153-46fc-adb3-ff09dbe153d1\") " pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:19.814881 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:19.814797 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:20.159753 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.159718 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j"] Sep 08 16:27:20.166812 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:27:20.166783 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a13e4b9_6153_46fc_adb3_ff09dbe153d1.slice/crio-9386f9943636d26b85d6210e07d6806e65f88c09b451012b19a7c9dd97356ee3 WatchSource:0}: Error finding container 9386f9943636d26b85d6210e07d6806e65f88c09b451012b19a7c9dd97356ee3: Status 404 returned error can't find the container with id 9386f9943636d26b85d6210e07d6806e65f88c09b451012b19a7c9dd97356ee3 Sep 08 16:27:20.245165 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.245137 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 16:27:20.393159 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:27:20.393128 2567 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Sep 08 16:27:20.393291 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:27:20.393205 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-apiservice-cert podName:6bf0430c-4800-48e9-925b-55bf8fae9ff8 nodeName:}" failed. No retries permitted until 2026-09-08 16:27:20.893188863 +0000 UTC m=+268.364175311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "apiservice-cert" (UniqueName: "kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-apiservice-cert") pod "opendatahub-operator-controller-manager-6fcf9b86-lwglg" (UID: "6bf0430c-4800-48e9-925b-55bf8fae9ff8") : failed to sync secret cache: timed out waiting for the condition Sep 08 16:27:20.393291 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:27:20.393131 2567 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Sep 08 16:27:20.393291 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:27:20.393281 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-webhook-cert podName:6bf0430c-4800-48e9-925b-55bf8fae9ff8 nodeName:}" failed. No retries permitted until 2026-09-08 16:27:20.893264623 +0000 UTC m=+268.364251060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-webhook-cert") pod "opendatahub-operator-controller-manager-6fcf9b86-lwglg" (UID: "6bf0430c-4800-48e9-925b-55bf8fae9ff8") : failed to sync secret cache: timed out waiting for the condition Sep 08 16:27:20.439474 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.439393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" event={"ID":"8a13e4b9-6153-46fc-adb3-ff09dbe153d1","Type":"ContainerStarted","Data":"9386f9943636d26b85d6210e07d6806e65f88c09b451012b19a7c9dd97356ee3"} Sep 08 16:27:20.724206 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.724105 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 16:27:20.778160 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.778128 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mtxq4\"" Sep 08 16:27:20.838011 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.837982 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 16:27:20.850584 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.850558 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz6tn\" (UniqueName: \"kubernetes.io/projected/6bf0430c-4800-48e9-925b-55bf8fae9ff8-kube-api-access-jz6tn\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:20.906958 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.906928 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:20.907113 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.907011 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:20.909427 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.909397 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:20.909595 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.909571 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bf0430c-4800-48e9-925b-55bf8fae9ff8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fcf9b86-lwglg\" (UID: \"6bf0430c-4800-48e9-925b-55bf8fae9ff8\") " pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:20.910479 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:20.910464 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 16:27:21.064095 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:21.064005 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:21.435313 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:27:21.435205 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf0430c_4800_48e9_925b_55bf8fae9ff8.slice/crio-42f7b1795c86e1307d2f83d3bcb4b8316c775a20a5f9558487825aa6905cf0f7 WatchSource:0}: Error finding container 42f7b1795c86e1307d2f83d3bcb4b8316c775a20a5f9558487825aa6905cf0f7: Status 404 returned error can't find the container with id 42f7b1795c86e1307d2f83d3bcb4b8316c775a20a5f9558487825aa6905cf0f7 Sep 08 16:27:21.437853 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:21.437588 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg"] Sep 08 16:27:21.447503 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:21.447477 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" event={"ID":"6bf0430c-4800-48e9-925b-55bf8fae9ff8","Type":"ContainerStarted","Data":"42f7b1795c86e1307d2f83d3bcb4b8316c775a20a5f9558487825aa6905cf0f7"} Sep 08 16:27:25.474981 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:25.474945 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" event={"ID":"6bf0430c-4800-48e9-925b-55bf8fae9ff8","Type":"ContainerStarted","Data":"240cd0b1856459330a25cc0d3cd19d52568c8140d76f4e4575a3d0dbdb0acd39"} Sep 08 16:27:25.476739 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:25.476711 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" event={"ID":"8a13e4b9-6153-46fc-adb3-ff09dbe153d1","Type":"ContainerStarted","Data":"cbab2fa8f011801eb498e2a110cac789c7b87c99f915681472019ded8e9ebf7f"} Sep 08 16:27:25.476877 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:25.476855 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:25.565908 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:25.565855 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" podStartSLOduration=1.394899913 podStartE2EDuration="6.56584091s" totalImagesPullingTime="5.170940997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:27:25.562625293 +0000 UTC m=+273.033611750" watchObservedRunningTime="2026-09-08 16:27:25.56584091 +0000 UTC m=+273.036827366" Sep 08 16:27:27.488413 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:27.488383 2567 generic.go:356] "Generic (PLEG): container finished" podID="6bf0430c-4800-48e9-925b-55bf8fae9ff8" containerID="240cd0b1856459330a25cc0d3cd19d52568c8140d76f4e4575a3d0dbdb0acd39" exitCode=0 Sep 08 16:27:27.488784 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:27.488419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" event={"ID":"6bf0430c-4800-48e9-925b-55bf8fae9ff8","Type":"ContainerDied","Data":"240cd0b1856459330a25cc0d3cd19d52568c8140d76f4e4575a3d0dbdb0acd39"} Sep 08 16:27:28.495806 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:28.495767 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" event={"ID":"6bf0430c-4800-48e9-925b-55bf8fae9ff8","Type":"ContainerStarted","Data":"2df247e2d51d794af19bded614eb81edc7d52679cb12c4ea5fdd766371e53f22"} Sep 08 16:27:28.496192 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:28.496003 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:28.542259 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:28.542204 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" podStartSLOduration=4.061895483 podStartE2EDuration="9.542187374s" totalImagesPullingTime="4.433383878s" totalInitContainerRuntime="1.046908013s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:27:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:27:28.540603641 +0000 UTC m=+276.011590120" watchObservedRunningTime="2026-09-08 16:27:28.542187374 +0000 UTC m=+276.013173829" Sep 08 16:27:36.484338 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:36.484304 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5ddb9f5cb8-lkz9j" Sep 08 16:27:39.502445 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:39.502415 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6fcf9b86-lwglg" Sep 08 16:27:43.056368 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.056335 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq"] Sep 08 16:27:43.072182 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.072154 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.077123 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.077077 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 08 16:27:43.077265 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.077249 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7zk2z\"" Sep 08 16:27:43.078014 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.077997 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 08 16:27:43.090796 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.090775 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 16:27:43.091578 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.091564 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 16:27:43.127732 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.127702 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq"] Sep 08 16:27:43.204077 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.204040 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f03454b1-e2b6-4fcc-927d-b47cb1f0a899-tmp\") pod \"kube-auth-proxy-56cf6b7cfc-nlzbq\" (UID: \"f03454b1-e2b6-4fcc-927d-b47cb1f0a899\") " pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.204502 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.204472 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03454b1-e2b6-4fcc-927d-b47cb1f0a899-tls-certs\") pod \"kube-auth-proxy-56cf6b7cfc-nlzbq\" (UID: \"f03454b1-e2b6-4fcc-927d-b47cb1f0a899\") " pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.204803 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.204765 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrlh\" (UniqueName: \"kubernetes.io/projected/f03454b1-e2b6-4fcc-927d-b47cb1f0a899-kube-api-access-zsrlh\") pod \"kube-auth-proxy-56cf6b7cfc-nlzbq\" (UID: \"f03454b1-e2b6-4fcc-927d-b47cb1f0a899\") " pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.305858 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.305815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03454b1-e2b6-4fcc-927d-b47cb1f0a899-tls-certs\") pod \"kube-auth-proxy-56cf6b7cfc-nlzbq\" (UID: \"f03454b1-e2b6-4fcc-927d-b47cb1f0a899\") " pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.306054 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.305876 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrlh\" (UniqueName: \"kubernetes.io/projected/f03454b1-e2b6-4fcc-927d-b47cb1f0a899-kube-api-access-zsrlh\") pod \"kube-auth-proxy-56cf6b7cfc-nlzbq\" (UID: \"f03454b1-e2b6-4fcc-927d-b47cb1f0a899\") " pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.306054 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.305922 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f03454b1-e2b6-4fcc-927d-b47cb1f0a899-tmp\") pod \"kube-auth-proxy-56cf6b7cfc-nlzbq\" (UID: \"f03454b1-e2b6-4fcc-927d-b47cb1f0a899\") " pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.308312 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.308257 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f03454b1-e2b6-4fcc-927d-b47cb1f0a899-tmp\") pod \"kube-auth-proxy-56cf6b7cfc-nlzbq\" (UID: \"f03454b1-e2b6-4fcc-927d-b47cb1f0a899\") " pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.308403 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.308389 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03454b1-e2b6-4fcc-927d-b47cb1f0a899-tls-certs\") pod \"kube-auth-proxy-56cf6b7cfc-nlzbq\" (UID: \"f03454b1-e2b6-4fcc-927d-b47cb1f0a899\") " pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.379145 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.379115 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrlh\" (UniqueName: \"kubernetes.io/projected/f03454b1-e2b6-4fcc-927d-b47cb1f0a899-kube-api-access-zsrlh\") pod \"kube-auth-proxy-56cf6b7cfc-nlzbq\" (UID: \"f03454b1-e2b6-4fcc-927d-b47cb1f0a899\") " pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.381890 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.381873 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" Sep 08 16:27:43.678799 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:43.678761 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq"] Sep 08 16:27:43.685029 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:27:43.684993 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03454b1_e2b6_4fcc_927d_b47cb1f0a899.slice/crio-df23a6bed701bb7127b92c905c76568e4532516f81c4e852554c3e506fa1e2fa WatchSource:0}: Error finding container df23a6bed701bb7127b92c905c76568e4532516f81c4e852554c3e506fa1e2fa: Status 404 returned error can't find the container with id df23a6bed701bb7127b92c905c76568e4532516f81c4e852554c3e506fa1e2fa Sep 08 16:27:44.572872 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:44.572419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" event={"ID":"f03454b1-e2b6-4fcc-927d-b47cb1f0a899","Type":"ContainerStarted","Data":"df23a6bed701bb7127b92c905c76568e4532516f81c4e852554c3e506fa1e2fa"} Sep 08 16:27:47.589146 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:47.589109 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" event={"ID":"f03454b1-e2b6-4fcc-927d-b47cb1f0a899","Type":"ContainerStarted","Data":"65d1d92bc6a303227285d13f3a185b4635e6d0aa4989e2866af3643d7f414253"} Sep 08 16:27:47.616773 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:47.616718 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-56cf6b7cfc-nlzbq" podStartSLOduration=2.751394497 podStartE2EDuration="5.61670264s" totalImagesPullingTime="2.865308143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:27:47.61578794 +0000 UTC m=+295.086774398" watchObservedRunningTime="2026-09-08 16:27:47.61670264 +0000 UTC m=+295.087689097" Sep 08 16:27:52.922138 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:52.922103 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:27:52.926730 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:52.926673 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:27:52.931291 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:27:52.931270 2567 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 16:29:22.355407 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.355368 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4smmm"] Sep 08 16:29:22.357678 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.357656 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" Sep 08 16:29:22.360676 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.360656 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 16:29:22.360772 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.360724 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 16:29:22.360954 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.360939 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dghfl\"" Sep 08 16:29:22.381721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.381699 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4smmm"] Sep 08 16:29:22.473090 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.473050 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkrx\" (UniqueName: \"kubernetes.io/projected/38899c90-da56-42ba-9d43-1958658bd59e-kube-api-access-plkrx\") pod \"authorino-operator-76b478bd78-4smmm\" (UID: \"38899c90-da56-42ba-9d43-1958658bd59e\") " pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" Sep 08 16:29:22.573903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.573864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkrx\" (UniqueName: \"kubernetes.io/projected/38899c90-da56-42ba-9d43-1958658bd59e-kube-api-access-plkrx\") pod \"authorino-operator-76b478bd78-4smmm\" (UID: \"38899c90-da56-42ba-9d43-1958658bd59e\") " pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" Sep 08 16:29:22.588507 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.588481 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkrx\" (UniqueName: \"kubernetes.io/projected/38899c90-da56-42ba-9d43-1958658bd59e-kube-api-access-plkrx\") pod \"authorino-operator-76b478bd78-4smmm\" (UID: \"38899c90-da56-42ba-9d43-1958658bd59e\") " pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" Sep 08 16:29:22.667255 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.667230 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" Sep 08 16:29:22.950893 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.950868 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4smmm"] Sep 08 16:29:22.953730 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:29:22.953690 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38899c90_da56_42ba_9d43_1958658bd59e.slice/crio-0a5e9018a8a2b21ad59f913c1fa8b8a4624e14d7867f1fdd974b008c59c133fc WatchSource:0}: Error finding container 0a5e9018a8a2b21ad59f913c1fa8b8a4624e14d7867f1fdd974b008c59c133fc: Status 404 returned error can't find the container with id 0a5e9018a8a2b21ad59f913c1fa8b8a4624e14d7867f1fdd974b008c59c133fc Sep 08 16:29:22.956036 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:22.955999 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:29:23.018002 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:23.017965 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" event={"ID":"38899c90-da56-42ba-9d43-1958658bd59e","Type":"ContainerStarted","Data":"0a5e9018a8a2b21ad59f913c1fa8b8a4624e14d7867f1fdd974b008c59c133fc"} Sep 08 16:29:26.034628 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:26.034585 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" event={"ID":"38899c90-da56-42ba-9d43-1958658bd59e","Type":"ContainerStarted","Data":"ad6a4f1bd8f9dd67669b872f5bdfa95ea77aec9c4654eaa2c9276589049ae0e2"} Sep 08 16:29:26.035109 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:26.034699 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" Sep 08 16:29:37.041801 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:37.041770 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" Sep 08 16:29:37.083345 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:29:37.083293 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-4smmm" podStartSLOduration=12.989116683 podStartE2EDuration="15.083276874s" totalImagesPullingTime="2.094160191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:26.061497862 +0000 UTC m=+393.532484317" watchObservedRunningTime="2026-09-08 16:29:37.083276874 +0000 UTC m=+404.554263330" Sep 08 16:30:13.292631 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.292597 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pvdbh"] Sep 08 16:30:13.300194 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.300172 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" Sep 08 16:30:13.306207 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.306186 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h5n8s\"" Sep 08 16:30:13.315610 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.315585 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pvdbh"] Sep 08 16:30:13.433391 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.433349 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jskqm\" (UniqueName: \"kubernetes.io/projected/ecc33bb9-41f8-4324-96e9-9f94ab71c8e1-kube-api-access-jskqm\") pod \"authorino-7bcfd8f9cd-pvdbh\" (UID: \"ecc33bb9-41f8-4324-96e9-9f94ab71c8e1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" Sep 08 16:30:13.534753 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.534718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jskqm\" (UniqueName: \"kubernetes.io/projected/ecc33bb9-41f8-4324-96e9-9f94ab71c8e1-kube-api-access-jskqm\") pod \"authorino-7bcfd8f9cd-pvdbh\" (UID: \"ecc33bb9-41f8-4324-96e9-9f94ab71c8e1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" Sep 08 16:30:13.547720 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.547647 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jskqm\" (UniqueName: \"kubernetes.io/projected/ecc33bb9-41f8-4324-96e9-9f94ab71c8e1-kube-api-access-jskqm\") pod \"authorino-7bcfd8f9cd-pvdbh\" (UID: \"ecc33bb9-41f8-4324-96e9-9f94ab71c8e1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" Sep 08 16:30:13.555604 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.555582 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-5ghqg"] Sep 08 16:30:13.559241 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.559223 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5ghqg" Sep 08 16:30:13.568770 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.568749 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5ghqg"] Sep 08 16:30:13.609736 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.609708 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" Sep 08 16:30:13.736736 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.736689 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ksv4\" (UniqueName: \"kubernetes.io/projected/f3b1fe5d-ca37-453c-bb26-341b7ac25c1e-kube-api-access-4ksv4\") pod \"authorino-59c974bd48-5ghqg\" (UID: \"f3b1fe5d-ca37-453c-bb26-341b7ac25c1e\") " pod="kuadrant-system/authorino-59c974bd48-5ghqg" Sep 08 16:30:13.837900 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.837810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ksv4\" (UniqueName: \"kubernetes.io/projected/f3b1fe5d-ca37-453c-bb26-341b7ac25c1e-kube-api-access-4ksv4\") pod \"authorino-59c974bd48-5ghqg\" (UID: \"f3b1fe5d-ca37-453c-bb26-341b7ac25c1e\") " pod="kuadrant-system/authorino-59c974bd48-5ghqg" Sep 08 16:30:13.847191 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.847160 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ksv4\" (UniqueName: \"kubernetes.io/projected/f3b1fe5d-ca37-453c-bb26-341b7ac25c1e-kube-api-access-4ksv4\") pod \"authorino-59c974bd48-5ghqg\" (UID: \"f3b1fe5d-ca37-453c-bb26-341b7ac25c1e\") " pod="kuadrant-system/authorino-59c974bd48-5ghqg" Sep 08 16:30:13.869072 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.869044 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5ghqg" Sep 08 16:30:13.879167 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:13.879143 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pvdbh"] Sep 08 16:30:13.884744 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:30:13.884712 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecc33bb9_41f8_4324_96e9_9f94ab71c8e1.slice/crio-f2a21d65a3113c737c1c06afe24f1a158618f1b736e3f2aa4b5ff715b0e76dcd WatchSource:0}: Error finding container f2a21d65a3113c737c1c06afe24f1a158618f1b736e3f2aa4b5ff715b0e76dcd: Status 404 returned error can't find the container with id f2a21d65a3113c737c1c06afe24f1a158618f1b736e3f2aa4b5ff715b0e76dcd Sep 08 16:30:14.148767 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:14.148744 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5ghqg"] Sep 08 16:30:14.151205 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:30:14.151178 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3b1fe5d_ca37_453c_bb26_341b7ac25c1e.slice/crio-0396ac6fd45848cfd089dc64055304bedbba0eac8e604aaa49096fca87da17ce WatchSource:0}: Error finding container 0396ac6fd45848cfd089dc64055304bedbba0eac8e604aaa49096fca87da17ce: Status 404 returned error can't find the container with id 0396ac6fd45848cfd089dc64055304bedbba0eac8e604aaa49096fca87da17ce Sep 08 16:30:14.263378 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:14.263343 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5ghqg" event={"ID":"f3b1fe5d-ca37-453c-bb26-341b7ac25c1e","Type":"ContainerStarted","Data":"0396ac6fd45848cfd089dc64055304bedbba0eac8e604aaa49096fca87da17ce"} Sep 08 16:30:14.264542 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:14.264507 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" event={"ID":"ecc33bb9-41f8-4324-96e9-9f94ab71c8e1","Type":"ContainerStarted","Data":"f2a21d65a3113c737c1c06afe24f1a158618f1b736e3f2aa4b5ff715b0e76dcd"} Sep 08 16:30:16.276045 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:16.275933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5ghqg" event={"ID":"f3b1fe5d-ca37-453c-bb26-341b7ac25c1e","Type":"ContainerStarted","Data":"946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61"} Sep 08 16:30:16.277409 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:16.277383 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" event={"ID":"ecc33bb9-41f8-4324-96e9-9f94ab71c8e1","Type":"ContainerStarted","Data":"830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3"} Sep 08 16:30:16.294897 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:16.294856 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-5ghqg" podStartSLOduration=1.507082198 podStartE2EDuration="3.29484013s" totalImagesPullingTime="1.787757932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:16.294144318 +0000 UTC m=+443.765130773" watchObservedRunningTime="2026-09-08 16:30:16.29484013 +0000 UTC m=+443.765826586" Sep 08 16:30:16.310965 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:16.310922 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" podStartSLOduration=1.261680256 podStartE2EDuration="3.310907151s" totalImagesPullingTime="2.049226895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:16.310160909 +0000 UTC m=+443.781147364" watchObservedRunningTime="2026-09-08 16:30:16.310907151 +0000 UTC m=+443.781893607" Sep 08 16:30:16.341111 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:16.341075 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pvdbh"] Sep 08 16:30:18.287583 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:18.287541 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" podUID="ecc33bb9-41f8-4324-96e9-9f94ab71c8e1" containerName="authorino" containerID="cri-o://830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3" gracePeriod=30 Sep 08 16:30:18.529194 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:18.529159 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" Sep 08 16:30:18.684827 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:18.684780 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jskqm\" (UniqueName: \"kubernetes.io/projected/ecc33bb9-41f8-4324-96e9-9f94ab71c8e1-kube-api-access-jskqm\") pod \"ecc33bb9-41f8-4324-96e9-9f94ab71c8e1\" (UID: \"ecc33bb9-41f8-4324-96e9-9f94ab71c8e1\") " Sep 08 16:30:18.687140 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:18.687108 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc33bb9-41f8-4324-96e9-9f94ab71c8e1-kube-api-access-jskqm" (OuterVolumeSpecName: "kube-api-access-jskqm") pod "ecc33bb9-41f8-4324-96e9-9f94ab71c8e1" (UID: "ecc33bb9-41f8-4324-96e9-9f94ab71c8e1"). InnerVolumeSpecName "kube-api-access-jskqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:30:18.785576 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:18.785530 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jskqm\" (UniqueName: \"kubernetes.io/projected/ecc33bb9-41f8-4324-96e9-9f94ab71c8e1-kube-api-access-jskqm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:30:19.293776 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:19.293734 2567 generic.go:356] "Generic (PLEG): container finished" podID="ecc33bb9-41f8-4324-96e9-9f94ab71c8e1" containerID="830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3" exitCode=0 Sep 08 16:30:19.293776 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:19.293766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" event={"ID":"ecc33bb9-41f8-4324-96e9-9f94ab71c8e1","Type":"ContainerDied","Data":"830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3"} Sep 08 16:30:19.294280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:19.293788 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" Sep 08 16:30:19.294280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:19.293810 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pvdbh" event={"ID":"ecc33bb9-41f8-4324-96e9-9f94ab71c8e1","Type":"ContainerDied","Data":"f2a21d65a3113c737c1c06afe24f1a158618f1b736e3f2aa4b5ff715b0e76dcd"} Sep 08 16:30:19.294280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:19.293826 2567 scope.go:117] "RemoveContainer" containerID="830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3" Sep 08 16:30:19.301963 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:19.301947 2567 scope.go:117] "RemoveContainer" containerID="830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3" Sep 08 16:30:19.302273 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:30:19.302256 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3\": container with ID starting with 830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3 not found: ID does not exist" containerID="830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3" Sep 08 16:30:19.302335 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:19.302280 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3"} err="failed to get container status \"830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3\": rpc error: code = NotFound desc = could not find container \"830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3\": container with ID starting with 830f4d2f193a8d3c3f2adfd4f7c497387d7b65aa67669a6c017bde09897817c3 not found: ID does not exist" Sep 08 16:30:19.313161 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:19.313099 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pvdbh"] Sep 08 16:30:19.317397 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:19.317377 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pvdbh"] Sep 08 16:30:21.052365 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:21.052331 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc33bb9-41f8-4324-96e9-9f94ab71c8e1" path="/var/lib/kubelet/pods/ecc33bb9-41f8-4324-96e9-9f94ab71c8e1/volumes" Sep 08 16:30:41.405726 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.405692 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-dc46j"] Sep 08 16:30:41.406143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.406097 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc33bb9-41f8-4324-96e9-9f94ab71c8e1" containerName="authorino" Sep 08 16:30:41.406143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.406110 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc33bb9-41f8-4324-96e9-9f94ab71c8e1" containerName="authorino" Sep 08 16:30:41.406214 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.406168 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc33bb9-41f8-4324-96e9-9f94ab71c8e1" containerName="authorino" Sep 08 16:30:41.410880 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.410864 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dc46j" Sep 08 16:30:41.419175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.419152 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-dc46j"] Sep 08 16:30:41.488476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.488446 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfv95\" (UniqueName: \"kubernetes.io/projected/fcd27415-d6b9-460c-804f-3634ce87c1a5-kube-api-access-wfv95\") pod \"authorino-557f876f69-dc46j\" (UID: \"fcd27415-d6b9-460c-804f-3634ce87c1a5\") " pod="kuadrant-system/authorino-557f876f69-dc46j" Sep 08 16:30:41.491436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.491410 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dc46j"] Sep 08 16:30:41.491701 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:30:41.491683 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wfv95], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-dc46j" podUID="fcd27415-d6b9-460c-804f-3634ce87c1a5" Sep 08 16:30:41.585371 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.585342 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-zczdz"] Sep 08 16:30:41.588817 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.588802 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:41.589033 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.588997 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfv95\" (UniqueName: \"kubernetes.io/projected/fcd27415-d6b9-460c-804f-3634ce87c1a5-kube-api-access-wfv95\") pod \"authorino-557f876f69-dc46j\" (UID: \"fcd27415-d6b9-460c-804f-3634ce87c1a5\") " pod="kuadrant-system/authorino-557f876f69-dc46j" Sep 08 16:30:41.591739 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.591718 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 08 16:30:41.602915 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.602893 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfv95\" (UniqueName: \"kubernetes.io/projected/fcd27415-d6b9-460c-804f-3634ce87c1a5-kube-api-access-wfv95\") pod \"authorino-557f876f69-dc46j\" (UID: \"fcd27415-d6b9-460c-804f-3634ce87c1a5\") " pod="kuadrant-system/authorino-557f876f69-dc46j" Sep 08 16:30:41.626862 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.626834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-zczdz"] Sep 08 16:30:41.690605 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.690534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4719927d-c38f-4100-97a5-fb84020a142c-tls-cert\") pod \"authorino-6d4dfc649f-zczdz\" (UID: \"4719927d-c38f-4100-97a5-fb84020a142c\") " pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:41.690745 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.690674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gprl\" (UniqueName: \"kubernetes.io/projected/4719927d-c38f-4100-97a5-fb84020a142c-kube-api-access-7gprl\") pod \"authorino-6d4dfc649f-zczdz\" (UID: \"4719927d-c38f-4100-97a5-fb84020a142c\") " pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:41.720028 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.719998 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-zczdz"] Sep 08 16:30:41.720339 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:30:41.720320 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7gprl tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-zczdz" podUID="4719927d-c38f-4100-97a5-fb84020a142c" Sep 08 16:30:41.780117 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.780083 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d89f6df6d-mggg7"] Sep 08 16:30:41.785285 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.785258 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:30:41.791387 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.791360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gprl\" (UniqueName: \"kubernetes.io/projected/4719927d-c38f-4100-97a5-fb84020a142c-kube-api-access-7gprl\") pod \"authorino-6d4dfc649f-zczdz\" (UID: \"4719927d-c38f-4100-97a5-fb84020a142c\") " pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:41.791513 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.791417 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4719927d-c38f-4100-97a5-fb84020a142c-tls-cert\") pod \"authorino-6d4dfc649f-zczdz\" (UID: \"4719927d-c38f-4100-97a5-fb84020a142c\") " pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:41.794167 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.794146 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4719927d-c38f-4100-97a5-fb84020a142c-tls-cert\") pod \"authorino-6d4dfc649f-zczdz\" (UID: \"4719927d-c38f-4100-97a5-fb84020a142c\") " pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:41.803415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.803394 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d89f6df6d-mggg7"] Sep 08 16:30:41.807151 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.807128 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gprl\" (UniqueName: \"kubernetes.io/projected/4719927d-c38f-4100-97a5-fb84020a142c-kube-api-access-7gprl\") pod \"authorino-6d4dfc649f-zczdz\" (UID: \"4719927d-c38f-4100-97a5-fb84020a142c\") " pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:41.892505 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.892471 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d21f38b5-a524-428a-b2ab-1cd462ca3300-tls-cert\") pod \"authorino-6d89f6df6d-mggg7\" (UID: \"d21f38b5-a524-428a-b2ab-1cd462ca3300\") " pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:30:41.892672 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.892526 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxfj\" (UniqueName: \"kubernetes.io/projected/d21f38b5-a524-428a-b2ab-1cd462ca3300-kube-api-access-xmxfj\") pod \"authorino-6d89f6df6d-mggg7\" (UID: \"d21f38b5-a524-428a-b2ab-1cd462ca3300\") " pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:30:41.993420 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.993330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d21f38b5-a524-428a-b2ab-1cd462ca3300-tls-cert\") pod \"authorino-6d89f6df6d-mggg7\" (UID: \"d21f38b5-a524-428a-b2ab-1cd462ca3300\") " pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:30:41.993420 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.993404 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxfj\" (UniqueName: \"kubernetes.io/projected/d21f38b5-a524-428a-b2ab-1cd462ca3300-kube-api-access-xmxfj\") pod \"authorino-6d89f6df6d-mggg7\" (UID: \"d21f38b5-a524-428a-b2ab-1cd462ca3300\") " pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:30:41.995877 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:41.995856 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d21f38b5-a524-428a-b2ab-1cd462ca3300-tls-cert\") pod \"authorino-6d89f6df6d-mggg7\" (UID: \"d21f38b5-a524-428a-b2ab-1cd462ca3300\") " pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:30:42.002298 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.002262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxfj\" (UniqueName: \"kubernetes.io/projected/d21f38b5-a524-428a-b2ab-1cd462ca3300-kube-api-access-xmxfj\") pod \"authorino-6d89f6df6d-mggg7\" (UID: \"d21f38b5-a524-428a-b2ab-1cd462ca3300\") " pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:30:42.095389 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.095347 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:30:42.369704 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.369610 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d89f6df6d-mggg7"] Sep 08 16:30:42.372807 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:30:42.372781 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd21f38b5_a524_428a_b2ab_1cd462ca3300.slice/crio-8c7f477dc365a029c4582a1ef2c40866f9e655e4ce97a050f4e4f7d5cf292a8b WatchSource:0}: Error finding container 8c7f477dc365a029c4582a1ef2c40866f9e655e4ce97a050f4e4f7d5cf292a8b: Status 404 returned error can't find the container with id 8c7f477dc365a029c4582a1ef2c40866f9e655e4ce97a050f4e4f7d5cf292a8b Sep 08 16:30:42.399845 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.399814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" event={"ID":"d21f38b5-a524-428a-b2ab-1cd462ca3300","Type":"ContainerStarted","Data":"8c7f477dc365a029c4582a1ef2c40866f9e655e4ce97a050f4e4f7d5cf292a8b"} Sep 08 16:30:42.399937 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.399847 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:42.399937 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.399861 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dc46j" Sep 08 16:30:42.407982 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.407967 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:42.409980 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.409965 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dc46j" Sep 08 16:30:42.498555 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.498521 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4719927d-c38f-4100-97a5-fb84020a142c-tls-cert\") pod \"4719927d-c38f-4100-97a5-fb84020a142c\" (UID: \"4719927d-c38f-4100-97a5-fb84020a142c\") " Sep 08 16:30:42.498722 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.498624 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfv95\" (UniqueName: \"kubernetes.io/projected/fcd27415-d6b9-460c-804f-3634ce87c1a5-kube-api-access-wfv95\") pod \"fcd27415-d6b9-460c-804f-3634ce87c1a5\" (UID: \"fcd27415-d6b9-460c-804f-3634ce87c1a5\") " Sep 08 16:30:42.498722 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.498665 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gprl\" (UniqueName: \"kubernetes.io/projected/4719927d-c38f-4100-97a5-fb84020a142c-kube-api-access-7gprl\") pod \"4719927d-c38f-4100-97a5-fb84020a142c\" (UID: \"4719927d-c38f-4100-97a5-fb84020a142c\") " Sep 08 16:30:42.500821 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.500796 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4719927d-c38f-4100-97a5-fb84020a142c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4719927d-c38f-4100-97a5-fb84020a142c" (UID: "4719927d-c38f-4100-97a5-fb84020a142c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:30:42.500919 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.500835 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4719927d-c38f-4100-97a5-fb84020a142c-kube-api-access-7gprl" (OuterVolumeSpecName: "kube-api-access-7gprl") pod "4719927d-c38f-4100-97a5-fb84020a142c" (UID: "4719927d-c38f-4100-97a5-fb84020a142c"). InnerVolumeSpecName "kube-api-access-7gprl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:30:42.500919 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.500842 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd27415-d6b9-460c-804f-3634ce87c1a5-kube-api-access-wfv95" (OuterVolumeSpecName: "kube-api-access-wfv95") pod "fcd27415-d6b9-460c-804f-3634ce87c1a5" (UID: "fcd27415-d6b9-460c-804f-3634ce87c1a5"). InnerVolumeSpecName "kube-api-access-wfv95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:30:42.600067 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.600013 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfv95\" (UniqueName: \"kubernetes.io/projected/fcd27415-d6b9-460c-804f-3634ce87c1a5-kube-api-access-wfv95\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:30:42.600067 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.600060 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gprl\" (UniqueName: \"kubernetes.io/projected/4719927d-c38f-4100-97a5-fb84020a142c-kube-api-access-7gprl\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:30:42.600067 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:42.600069 2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4719927d-c38f-4100-97a5-fb84020a142c-tls-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:30:43.407089 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.407050 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" event={"ID":"d21f38b5-a524-428a-b2ab-1cd462ca3300","Type":"ContainerStarted","Data":"a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75"} Sep 08 16:30:43.407314 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.407189 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dc46j" Sep 08 16:30:43.407378 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.407323 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-zczdz" Sep 08 16:30:43.423779 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.423735 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" podStartSLOduration=1.864286039 podStartE2EDuration="2.423720902s" totalImagesPullingTime="559.434863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:43.423092884 +0000 UTC m=+470.894079340" watchObservedRunningTime="2026-09-08 16:30:43.423720902 +0000 UTC m=+470.894707358" Sep 08 16:30:43.450661 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.450633 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-zczdz"] Sep 08 16:30:43.456794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.456762 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5ghqg"] Sep 08 16:30:43.456975 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.456952 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-5ghqg" podUID="f3b1fe5d-ca37-453c-bb26-341b7ac25c1e" containerName="authorino" containerID="cri-o://946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61" gracePeriod=30 Sep 08 16:30:43.462742 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.462647 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-zczdz"] Sep 08 16:30:43.502320 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.502287 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dc46j"] Sep 08 16:30:43.510593 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.510561 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-dc46j"] Sep 08 16:30:43.698139 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.698113 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5ghqg" Sep 08 16:30:43.809470 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.809441 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ksv4\" (UniqueName: \"kubernetes.io/projected/f3b1fe5d-ca37-453c-bb26-341b7ac25c1e-kube-api-access-4ksv4\") pod \"f3b1fe5d-ca37-453c-bb26-341b7ac25c1e\" (UID: \"f3b1fe5d-ca37-453c-bb26-341b7ac25c1e\") " Sep 08 16:30:43.811646 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.811620 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b1fe5d-ca37-453c-bb26-341b7ac25c1e-kube-api-access-4ksv4" (OuterVolumeSpecName: "kube-api-access-4ksv4") pod "f3b1fe5d-ca37-453c-bb26-341b7ac25c1e" (UID: "f3b1fe5d-ca37-453c-bb26-341b7ac25c1e"). InnerVolumeSpecName "kube-api-access-4ksv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:30:43.911009 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:43.910976 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ksv4\" (UniqueName: \"kubernetes.io/projected/f3b1fe5d-ca37-453c-bb26-341b7ac25c1e-kube-api-access-4ksv4\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:30:44.413358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:44.413326 2567 generic.go:356] "Generic (PLEG): container finished" podID="f3b1fe5d-ca37-453c-bb26-341b7ac25c1e" containerID="946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61" exitCode=0 Sep 08 16:30:44.413551 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:44.413371 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5ghqg" Sep 08 16:30:44.413551 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:44.413416 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5ghqg" event={"ID":"f3b1fe5d-ca37-453c-bb26-341b7ac25c1e","Type":"ContainerDied","Data":"946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61"} Sep 08 16:30:44.413551 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:44.413462 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5ghqg" event={"ID":"f3b1fe5d-ca37-453c-bb26-341b7ac25c1e","Type":"ContainerDied","Data":"0396ac6fd45848cfd089dc64055304bedbba0eac8e604aaa49096fca87da17ce"} Sep 08 16:30:44.413551 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:44.413488 2567 scope.go:117] "RemoveContainer" containerID="946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61" Sep 08 16:30:44.422149 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:44.422131 2567 scope.go:117] "RemoveContainer" containerID="946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61" Sep 08 16:30:44.422429 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:30:44.422412 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61\": container with ID starting with 946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61 not found: ID does not exist" containerID="946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61" Sep 08 16:30:44.422495 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:44.422435 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61"} err="failed to get container status \"946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61\": rpc error: code = NotFound desc = could not find container \"946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61\": container with ID starting with 946cf98798222726941bd4c4e060cf75a7510307fba7c53bfd5434c9db2e4a61 not found: ID does not exist" Sep 08 16:30:44.433851 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:44.433828 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5ghqg"] Sep 08 16:30:44.437285 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:44.437263 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5ghqg"] Sep 08 16:30:45.058693 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:45.058660 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4719927d-c38f-4100-97a5-fb84020a142c" path="/var/lib/kubelet/pods/4719927d-c38f-4100-97a5-fb84020a142c/volumes" Sep 08 16:30:45.058880 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:45.058868 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3b1fe5d-ca37-453c-bb26-341b7ac25c1e" path="/var/lib/kubelet/pods/f3b1fe5d-ca37-453c-bb26-341b7ac25c1e/volumes" Sep 08 16:30:45.059177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:30:45.059165 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcd27415-d6b9-460c-804f-3634ce87c1a5" path="/var/lib/kubelet/pods/fcd27415-d6b9-460c-804f-3634ce87c1a5/volumes" Sep 08 16:31:25.322294 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.322261 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf"] Sep 08 16:31:25.322874 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.322812 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b1fe5d-ca37-453c-bb26-341b7ac25c1e" containerName="authorino" Sep 08 16:31:25.322874 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.322830 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b1fe5d-ca37-453c-bb26-341b7ac25c1e" containerName="authorino" Sep 08 16:31:25.322992 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.322937 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b1fe5d-ca37-453c-bb26-341b7ac25c1e" containerName="authorino" Sep 08 16:31:25.325802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.325782 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:31:25.330562 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.330531 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 08 16:31:25.338476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.338454 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf"] Sep 08 16:31:25.378698 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.378659 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk8lp\" (UniqueName: \"kubernetes.io/projected/1f0a8495-338d-4e24-a0af-e4fc2f9f0be1-kube-api-access-pk8lp\") pod \"payload-pre-processing-59fbc47bf8-ltrmf\" (UID: \"1f0a8495-338d-4e24-a0af-e4fc2f9f0be1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:31:25.378872 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.378710 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0a8495-338d-4e24-a0af-e4fc2f9f0be1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-ltrmf\" (UID: \"1f0a8495-338d-4e24-a0af-e4fc2f9f0be1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:31:25.479547 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.479510 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk8lp\" (UniqueName: \"kubernetes.io/projected/1f0a8495-338d-4e24-a0af-e4fc2f9f0be1-kube-api-access-pk8lp\") pod \"payload-pre-processing-59fbc47bf8-ltrmf\" (UID: \"1f0a8495-338d-4e24-a0af-e4fc2f9f0be1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:31:25.479547 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.479563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0a8495-338d-4e24-a0af-e4fc2f9f0be1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-ltrmf\" (UID: \"1f0a8495-338d-4e24-a0af-e4fc2f9f0be1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:31:25.480261 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.480242 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0a8495-338d-4e24-a0af-e4fc2f9f0be1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-ltrmf\" (UID: \"1f0a8495-338d-4e24-a0af-e4fc2f9f0be1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:31:25.488794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.488759 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk8lp\" (UniqueName: \"kubernetes.io/projected/1f0a8495-338d-4e24-a0af-e4fc2f9f0be1-kube-api-access-pk8lp\") pod \"payload-pre-processing-59fbc47bf8-ltrmf\" (UID: \"1f0a8495-338d-4e24-a0af-e4fc2f9f0be1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:31:25.636594 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.636507 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:31:25.917506 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:25.917474 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf"] Sep 08 16:31:25.920764 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:31:25.920738 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0a8495_338d_4e24_a0af_e4fc2f9f0be1.slice/crio-176686072b95d1cc7f241f08124a2efa504f1b9368cac7684db660304183504c WatchSource:0}: Error finding container 176686072b95d1cc7f241f08124a2efa504f1b9368cac7684db660304183504c: Status 404 returned error can't find the container with id 176686072b95d1cc7f241f08124a2efa504f1b9368cac7684db660304183504c Sep 08 16:31:26.610669 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:26.610629 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" event={"ID":"1f0a8495-338d-4e24-a0af-e4fc2f9f0be1","Type":"ContainerStarted","Data":"176686072b95d1cc7f241f08124a2efa504f1b9368cac7684db660304183504c"} Sep 08 16:31:27.617840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:27.617796 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" event={"ID":"1f0a8495-338d-4e24-a0af-e4fc2f9f0be1","Type":"ContainerStarted","Data":"244e6b4b5600886560a875cebcd827fb80946a01f37589ac1a3756ae8ead030b"} Sep 08 16:31:27.618260 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:27.617946 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:31:27.636448 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:27.636391 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" podStartSLOduration=1.15784726 podStartE2EDuration="2.636370547s" totalImagesPullingTime="1.478523287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:27.634662882 +0000 UTC m=+515.105649339" watchObservedRunningTime="2026-09-08 16:31:27.636370547 +0000 UTC m=+515.107357007" Sep 08 16:31:38.625177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:31:38.625145 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ltrmf" Sep 08 16:32:20.752779 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.752740 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm"] Sep 08 16:32:20.756497 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.756480 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.759406 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.759388 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-whvrm\"" Sep 08 16:32:20.759500 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.759424 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 16:32:20.760112 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.760096 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 16:32:20.760210 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.760151 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 08 16:32:20.767924 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.767905 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm"] Sep 08 16:32:20.869212 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.869181 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8734e94-31dc-41fb-ba8c-b40fe657a174-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.869408 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.869230 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jnz\" (UniqueName: \"kubernetes.io/projected/f8734e94-31dc-41fb-ba8c-b40fe657a174-kube-api-access-j4jnz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.869408 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.869300 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.869408 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.869361 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.869408 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.869392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.869581 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.869424 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.970253 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.970216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.970395 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.970259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.970395 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.970299 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8734e94-31dc-41fb-ba8c-b40fe657a174-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.970395 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.970332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jnz\" (UniqueName: \"kubernetes.io/projected/f8734e94-31dc-41fb-ba8c-b40fe657a174-kube-api-access-j4jnz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.970395 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.970361 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.970395 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.970393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.970688 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.970665 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.970776 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.970755 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.970831 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.970810 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.972794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.972771 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8734e94-31dc-41fb-ba8c-b40fe657a174-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.973046 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.973009 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8734e94-31dc-41fb-ba8c-b40fe657a174-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:20.978163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:20.978141 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jnz\" (UniqueName: \"kubernetes.io/projected/f8734e94-31dc-41fb-ba8c-b40fe657a174-kube-api-access-j4jnz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm\" (UID: \"f8734e94-31dc-41fb-ba8c-b40fe657a174\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:21.067777 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:21.067691 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:21.340553 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:21.340525 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm"] Sep 08 16:32:21.343140 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:32:21.343100 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8734e94_31dc_41fb_ba8c_b40fe657a174.slice/crio-b63d6197afaaa4bc98eee182d4bb956a2b2f52958b1173f6e2fb7f59a8a883f6 WatchSource:0}: Error finding container b63d6197afaaa4bc98eee182d4bb956a2b2f52958b1173f6e2fb7f59a8a883f6: Status 404 returned error can't find the container with id b63d6197afaaa4bc98eee182d4bb956a2b2f52958b1173f6e2fb7f59a8a883f6 Sep 08 16:32:21.876969 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:21.876927 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" event={"ID":"f8734e94-31dc-41fb-ba8c-b40fe657a174","Type":"ContainerStarted","Data":"b63d6197afaaa4bc98eee182d4bb956a2b2f52958b1173f6e2fb7f59a8a883f6"} Sep 08 16:32:22.883892 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:22.883804 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" event={"ID":"f8734e94-31dc-41fb-ba8c-b40fe657a174","Type":"ContainerStarted","Data":"c9d50531fc28d82d34878b430ba1654777053ea9dd2454b79bdc30bdcb34f49b"} Sep 08 16:32:22.903765 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:22.903709 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" podStartSLOduration=1.827653454 podStartE2EDuration="2.903691981s" totalImagesPullingTime="1.076038527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:32:22.902763824 +0000 UTC m=+570.373750279" watchObservedRunningTime="2026-09-08 16:32:22.903691981 +0000 UTC m=+570.374678437" Sep 08 16:32:31.068226 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:31.068188 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:31.068710 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:31.068251 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:31.080976 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:31.080951 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:31.936579 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:31.936542 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm" Sep 08 16:32:52.957884 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:52.957855 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:32:52.958857 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:32:52.958834 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:33:02.156332 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.156299 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb"] Sep 08 16:33:02.161280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.161260 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.164194 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.164170 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 08 16:33:02.172744 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.172719 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb"] Sep 08 16:33:02.258503 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.258465 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.258683 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.258523 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.258683 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.258562 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.258683 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.258581 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.258683 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.258619 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm84b\" (UniqueName: \"kubernetes.io/projected/d710d22b-c6e8-4658-8249-81c800e5a2e0-kube-api-access-mm84b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.258818 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.258720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d710d22b-c6e8-4658-8249-81c800e5a2e0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.360100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.360054 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.360305 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.360109 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.360305 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.360133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.360305 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.360188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm84b\" (UniqueName: \"kubernetes.io/projected/d710d22b-c6e8-4658-8249-81c800e5a2e0-kube-api-access-mm84b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.360305 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.360256 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d710d22b-c6e8-4658-8249-81c800e5a2e0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.360305 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.360293 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.360582 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.360497 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.360582 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.360543 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.360582 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.360572 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.362717 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.362698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d710d22b-c6e8-4658-8249-81c800e5a2e0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.362821 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.362801 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d710d22b-c6e8-4658-8249-81c800e5a2e0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.369261 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.369226 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm84b\" (UniqueName: \"kubernetes.io/projected/d710d22b-c6e8-4658-8249-81c800e5a2e0-kube-api-access-mm84b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb\" (UID: \"d710d22b-c6e8-4658-8249-81c800e5a2e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.473313 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.473208 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:02.745531 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:02.745452 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb"] Sep 08 16:33:02.748286 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:33:02.748258 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd710d22b_c6e8_4658_8249_81c800e5a2e0.slice/crio-235caba2e8cd0c6a0223cda9f7e600eb59c4c8522daa1eb877ee05089fb8bfe5 WatchSource:0}: Error finding container 235caba2e8cd0c6a0223cda9f7e600eb59c4c8522daa1eb877ee05089fb8bfe5: Status 404 returned error can't find the container with id 235caba2e8cd0c6a0223cda9f7e600eb59c4c8522daa1eb877ee05089fb8bfe5 Sep 08 16:33:03.071925 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:03.071891 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" event={"ID":"d710d22b-c6e8-4658-8249-81c800e5a2e0","Type":"ContainerStarted","Data":"f030868cdfcff860174cd119afee6ec36a74e4cf5d303dda3f2efe6463f9afe3"} Sep 08 16:33:03.072123 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:03.071943 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" event={"ID":"d710d22b-c6e8-4658-8249-81c800e5a2e0","Type":"ContainerStarted","Data":"235caba2e8cd0c6a0223cda9f7e600eb59c4c8522daa1eb877ee05089fb8bfe5"} Sep 08 16:33:03.090292 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:03.090237 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" podStartSLOduration=0.888484074 podStartE2EDuration="1.090219253s" totalImagesPullingTime="201.735179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:03.088429967 +0000 UTC m=+610.559416447" watchObservedRunningTime="2026-09-08 16:33:03.090219253 +0000 UTC m=+610.561205709" Sep 08 16:33:12.473575 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:12.473537 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:12.474013 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:12.473605 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:12.486061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:12.486039 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:13.131158 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:13.131127 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb" Sep 08 16:33:24.859076 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.858974 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv"] Sep 08 16:33:24.862546 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.862530 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:24.865034 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.865006 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 08 16:33:24.872420 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.872399 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv"] Sep 08 16:33:24.977674 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.977632 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7ab6632-24a2-4d65-b78e-3e63f23216c0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:24.977875 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.977686 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:24.977875 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.977708 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:24.977875 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.977766 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt4f6\" (UniqueName: \"kubernetes.io/projected/d7ab6632-24a2-4d65-b78e-3e63f23216c0-kube-api-access-vt4f6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:24.977875 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.977811 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:24.977875 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:24.977855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.078939 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.078908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7ab6632-24a2-4d65-b78e-3e63f23216c0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.079156 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.078954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.079156 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.078977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.079156 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.078995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt4f6\" (UniqueName: \"kubernetes.io/projected/d7ab6632-24a2-4d65-b78e-3e63f23216c0-kube-api-access-vt4f6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.079156 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.079037 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.079156 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.079062 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.079469 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.079436 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.079469 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.079456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.079562 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.079482 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.081371 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.081348 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7ab6632-24a2-4d65-b78e-3e63f23216c0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.081789 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.081768 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7ab6632-24a2-4d65-b78e-3e63f23216c0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.088519 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.088496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt4f6\" (UniqueName: \"kubernetes.io/projected/d7ab6632-24a2-4d65-b78e-3e63f23216c0-kube-api-access-vt4f6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cfghv\" (UID: \"d7ab6632-24a2-4d65-b78e-3e63f23216c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.172648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.172608 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:25.447488 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:25.447460 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv"] Sep 08 16:33:26.194681 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:26.194649 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" event={"ID":"d7ab6632-24a2-4d65-b78e-3e63f23216c0","Type":"ContainerStarted","Data":"df68a99bbf546409e816f9d1a98bf84724f831dff6fe918f693805ad21189850"} Sep 08 16:33:26.195089 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:26.194689 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" event={"ID":"d7ab6632-24a2-4d65-b78e-3e63f23216c0","Type":"ContainerStarted","Data":"150ee84151db17f261fac034c46ed104e25b0541a0e81fb50af6c84b3a48f3b4"} Sep 08 16:33:26.212207 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:26.212149 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" podStartSLOduration=2.021016649 podStartE2EDuration="2.212133391s" totalImagesPullingTime="191.116742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:26.210272144 +0000 UTC m=+633.681258621" watchObservedRunningTime="2026-09-08 16:33:26.212133391 +0000 UTC m=+633.683119847" Sep 08 16:33:35.172799 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:35.172761 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:35.172799 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:35.172809 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:35.185318 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:35.185296 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:33:35.250403 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:33:35.250373 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cfghv" Sep 08 16:34:26.375162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.375123 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cbd679f45-jbrwq"] Sep 08 16:34:26.378539 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.378521 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cbd679f45-jbrwq" Sep 08 16:34:26.385507 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.385483 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cbd679f45-jbrwq"] Sep 08 16:34:26.438678 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.438638 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2be41c57-7cf8-4471-a627-a801a33f7588-tls-cert\") pod \"authorino-5cbd679f45-jbrwq\" (UID: \"2be41c57-7cf8-4471-a627-a801a33f7588\") " pod="kuadrant-system/authorino-5cbd679f45-jbrwq" Sep 08 16:34:26.438857 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.438748 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvj6j\" (UniqueName: \"kubernetes.io/projected/2be41c57-7cf8-4471-a627-a801a33f7588-kube-api-access-dvj6j\") pod \"authorino-5cbd679f45-jbrwq\" (UID: \"2be41c57-7cf8-4471-a627-a801a33f7588\") " pod="kuadrant-system/authorino-5cbd679f45-jbrwq" Sep 08 16:34:26.540274 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.540223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvj6j\" (UniqueName: \"kubernetes.io/projected/2be41c57-7cf8-4471-a627-a801a33f7588-kube-api-access-dvj6j\") pod \"authorino-5cbd679f45-jbrwq\" (UID: \"2be41c57-7cf8-4471-a627-a801a33f7588\") " pod="kuadrant-system/authorino-5cbd679f45-jbrwq" Sep 08 16:34:26.540482 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.540438 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2be41c57-7cf8-4471-a627-a801a33f7588-tls-cert\") pod \"authorino-5cbd679f45-jbrwq\" (UID: \"2be41c57-7cf8-4471-a627-a801a33f7588\") " pod="kuadrant-system/authorino-5cbd679f45-jbrwq" Sep 08 16:34:26.542984 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.542962 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2be41c57-7cf8-4471-a627-a801a33f7588-tls-cert\") pod \"authorino-5cbd679f45-jbrwq\" (UID: \"2be41c57-7cf8-4471-a627-a801a33f7588\") " pod="kuadrant-system/authorino-5cbd679f45-jbrwq" Sep 08 16:34:26.548627 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.548601 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvj6j\" (UniqueName: \"kubernetes.io/projected/2be41c57-7cf8-4471-a627-a801a33f7588-kube-api-access-dvj6j\") pod \"authorino-5cbd679f45-jbrwq\" (UID: \"2be41c57-7cf8-4471-a627-a801a33f7588\") " pod="kuadrant-system/authorino-5cbd679f45-jbrwq" Sep 08 16:34:26.689010 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.688981 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cbd679f45-jbrwq" Sep 08 16:34:26.960898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.960812 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cbd679f45-jbrwq"] Sep 08 16:34:26.967363 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:34:26.967299 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2be41c57_7cf8_4471_a627_a801a33f7588.slice/crio-6ecec6bb71b4298e5df05e23c64e966bb351bac7c7d3cdad011ed1bbe85b0ead WatchSource:0}: Error finding container 6ecec6bb71b4298e5df05e23c64e966bb351bac7c7d3cdad011ed1bbe85b0ead: Status 404 returned error can't find the container with id 6ecec6bb71b4298e5df05e23c64e966bb351bac7c7d3cdad011ed1bbe85b0ead Sep 08 16:34:26.968204 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:26.968180 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:34:27.499094 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:27.499058 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cbd679f45-jbrwq" event={"ID":"2be41c57-7cf8-4471-a627-a801a33f7588","Type":"ContainerStarted","Data":"6ecec6bb71b4298e5df05e23c64e966bb351bac7c7d3cdad011ed1bbe85b0ead"} Sep 08 16:34:28.507489 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:28.507447 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cbd679f45-jbrwq" event={"ID":"2be41c57-7cf8-4471-a627-a801a33f7588","Type":"ContainerStarted","Data":"a6e2a904b1eeae5c61c3ff2f389b303bf17c65aac0af989b8250ea14cfc3d734"} Sep 08 16:34:28.564771 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:28.564696 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5cbd679f45-jbrwq" podStartSLOduration=1.850325345 podStartE2EDuration="2.564673902s" totalImagesPullingTime="714.348557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:28.563264009 +0000 UTC m=+696.034250466" watchObservedRunningTime="2026-09-08 16:34:28.564673902 +0000 UTC m=+696.035660356" Sep 08 16:34:28.651897 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:28.651865 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d89f6df6d-mggg7"] Sep 08 16:34:28.652330 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:28.652301 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" podUID="d21f38b5-a524-428a-b2ab-1cd462ca3300" containerName="authorino" containerID="cri-o://a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75" gracePeriod=30 Sep 08 16:34:28.912340 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:28.912314 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:34:28.963409 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:28.963373 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmxfj\" (UniqueName: \"kubernetes.io/projected/d21f38b5-a524-428a-b2ab-1cd462ca3300-kube-api-access-xmxfj\") pod \"d21f38b5-a524-428a-b2ab-1cd462ca3300\" (UID: \"d21f38b5-a524-428a-b2ab-1cd462ca3300\") " Sep 08 16:34:28.963593 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:28.963414 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d21f38b5-a524-428a-b2ab-1cd462ca3300-tls-cert\") pod \"d21f38b5-a524-428a-b2ab-1cd462ca3300\" (UID: \"d21f38b5-a524-428a-b2ab-1cd462ca3300\") " Sep 08 16:34:28.965742 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:28.965713 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21f38b5-a524-428a-b2ab-1cd462ca3300-kube-api-access-xmxfj" (OuterVolumeSpecName: "kube-api-access-xmxfj") pod "d21f38b5-a524-428a-b2ab-1cd462ca3300" (UID: "d21f38b5-a524-428a-b2ab-1cd462ca3300"). InnerVolumeSpecName "kube-api-access-xmxfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:34:28.974253 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:28.974228 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21f38b5-a524-428a-b2ab-1cd462ca3300-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d21f38b5-a524-428a-b2ab-1cd462ca3300" (UID: "d21f38b5-a524-428a-b2ab-1cd462ca3300"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:34:29.069003 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.068944 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxfj\" (UniqueName: \"kubernetes.io/projected/d21f38b5-a524-428a-b2ab-1cd462ca3300-kube-api-access-xmxfj\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:34:29.069003 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.068974 2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d21f38b5-a524-428a-b2ab-1cd462ca3300-tls-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:34:29.513601 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.513566 2567 generic.go:356] "Generic (PLEG): container finished" podID="d21f38b5-a524-428a-b2ab-1cd462ca3300" containerID="a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75" exitCode=0 Sep 08 16:34:29.514051 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.513612 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" Sep 08 16:34:29.514051 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.513650 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" event={"ID":"d21f38b5-a524-428a-b2ab-1cd462ca3300","Type":"ContainerDied","Data":"a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75"} Sep 08 16:34:29.514051 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.513690 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d89f6df6d-mggg7" event={"ID":"d21f38b5-a524-428a-b2ab-1cd462ca3300","Type":"ContainerDied","Data":"8c7f477dc365a029c4582a1ef2c40866f9e655e4ce97a050f4e4f7d5cf292a8b"} Sep 08 16:34:29.514051 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.513705 2567 scope.go:117] "RemoveContainer" containerID="a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75" Sep 08 16:34:29.524656 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.524636 2567 scope.go:117] "RemoveContainer" containerID="a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75" Sep 08 16:34:29.525062 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:34:29.525032 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75\": container with ID starting with a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75 not found: ID does not exist" containerID="a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75" Sep 08 16:34:29.525180 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.525073 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75"} err="failed to get container status \"a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75\": rpc error: code = NotFound desc = could not find container \"a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75\": container with ID starting with a0729db562d8a323dfc9e3ab3e63e4ff3c5ba63a31a560ffbeb507745c3aed75 not found: ID does not exist" Sep 08 16:34:29.537779 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.537748 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d89f6df6d-mggg7"] Sep 08 16:34:29.547463 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:29.547436 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d89f6df6d-mggg7"] Sep 08 16:34:31.053260 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:31.053225 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d21f38b5-a524-428a-b2ab-1cd462ca3300" path="/var/lib/kubelet/pods/d21f38b5-a524-428a-b2ab-1cd462ca3300/volumes" Sep 08 16:34:48.375172 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.375094 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7851-140370438503216"] Sep 08 16:34:48.375522 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.375498 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21f38b5-a524-428a-b2ab-1cd462ca3300" containerName="authorino" Sep 08 16:34:48.375522 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.375511 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21f38b5-a524-428a-b2ab-1cd462ca3300" containerName="authorino" Sep 08 16:34:48.375594 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.375583 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21f38b5-a524-428a-b2ab-1cd462ca3300" containerName="authorino" Sep 08 16:34:48.378606 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.378590 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-140370438503216" Sep 08 16:34:48.383182 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.383164 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mrsv6\"" Sep 08 16:34:48.391453 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.391433 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-140370438503216"] Sep 08 16:34:48.549617 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.549574 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwt8\" (UniqueName: \"kubernetes.io/projected/bdc3faa9-e2c5-434a-a308-9482f1fc7719-kube-api-access-xpwt8\") pod \"test-curl-7851-140370438503216\" (UID: \"bdc3faa9-e2c5-434a-a308-9482f1fc7719\") " pod="opendatahub/test-curl-7851-140370438503216" Sep 08 16:34:48.650578 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.650494 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwt8\" (UniqueName: \"kubernetes.io/projected/bdc3faa9-e2c5-434a-a308-9482f1fc7719-kube-api-access-xpwt8\") pod \"test-curl-7851-140370438503216\" (UID: \"bdc3faa9-e2c5-434a-a308-9482f1fc7719\") " pod="opendatahub/test-curl-7851-140370438503216" Sep 08 16:34:48.659524 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.659505 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwt8\" (UniqueName: \"kubernetes.io/projected/bdc3faa9-e2c5-434a-a308-9482f1fc7719-kube-api-access-xpwt8\") pod \"test-curl-7851-140370438503216\" (UID: \"bdc3faa9-e2c5-434a-a308-9482f1fc7719\") " pod="opendatahub/test-curl-7851-140370438503216" Sep 08 16:34:48.687453 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.687423 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-140370438503216" Sep 08 16:34:48.957994 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:48.957968 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-140370438503216"] Sep 08 16:34:48.960383 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:34:48.960355 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdc3faa9_e2c5_434a_a308_9482f1fc7719.slice/crio-507547eae4cf7052374a1950f588a0f4fb959a99a3ad8a26bdf99a462bc67611 WatchSource:0}: Error finding container 507547eae4cf7052374a1950f588a0f4fb959a99a3ad8a26bdf99a462bc67611: Status 404 returned error can't find the container with id 507547eae4cf7052374a1950f588a0f4fb959a99a3ad8a26bdf99a462bc67611 Sep 08 16:34:49.619302 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:49.619272 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-140370438503216" event={"ID":"bdc3faa9-e2c5-434a-a308-9482f1fc7719","Type":"ContainerStarted","Data":"507547eae4cf7052374a1950f588a0f4fb959a99a3ad8a26bdf99a462bc67611"} Sep 08 16:34:50.626142 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:50.626102 2567 generic.go:356] "Generic (PLEG): container finished" podID="bdc3faa9-e2c5-434a-a308-9482f1fc7719" containerID="5d2eb331ef1fa9cae30a9d6c4d8a09c48fceca9d0d0681da82c5c6d2b5e8904c" exitCode=0 Sep 08 16:34:50.626502 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:50.626191 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-140370438503216" event={"ID":"bdc3faa9-e2c5-434a-a308-9482f1fc7719","Type":"ContainerDied","Data":"5d2eb331ef1fa9cae30a9d6c4d8a09c48fceca9d0d0681da82c5c6d2b5e8904c"} Sep 08 16:34:51.757913 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:51.757880 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-140370438503216" Sep 08 16:34:51.778805 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:51.778781 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpwt8\" (UniqueName: \"kubernetes.io/projected/bdc3faa9-e2c5-434a-a308-9482f1fc7719-kube-api-access-xpwt8\") pod \"bdc3faa9-e2c5-434a-a308-9482f1fc7719\" (UID: \"bdc3faa9-e2c5-434a-a308-9482f1fc7719\") " Sep 08 16:34:51.781034 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:51.780993 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc3faa9-e2c5-434a-a308-9482f1fc7719-kube-api-access-xpwt8" (OuterVolumeSpecName: "kube-api-access-xpwt8") pod "bdc3faa9-e2c5-434a-a308-9482f1fc7719" (UID: "bdc3faa9-e2c5-434a-a308-9482f1fc7719"). InnerVolumeSpecName "kube-api-access-xpwt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:34:51.848014 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:51.847984 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7851-140370438503216_bdc3faa9-e2c5-434a-a308-9482f1fc7719/test-curl-7851-140370438503216/0.log" Sep 08 16:34:51.879422 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:51.879384 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwt8\" (UniqueName: \"kubernetes.io/projected/bdc3faa9-e2c5-434a-a308-9482f1fc7719-kube-api-access-xpwt8\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:34:51.896457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:51.896388 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7851-140370438503216"] Sep 08 16:34:51.906354 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:51.906327 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7851-140370438503216"] Sep 08 16:34:52.292861 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.292826 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf"] Sep 08 16:34:52.293247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.293234 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc3faa9-e2c5-434a-a308-9482f1fc7719" containerName="test-curl-7851-140370438503216" Sep 08 16:34:52.293358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.293249 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc3faa9-e2c5-434a-a308-9482f1fc7719" containerName="test-curl-7851-140370438503216" Sep 08 16:34:52.293358 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.293319 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc3faa9-e2c5-434a-a308-9482f1fc7719" containerName="test-curl-7851-140370438503216" Sep 08 16:34:52.296495 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.296477 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:34:52.304308 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.304288 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ff43b8\"" Sep 08 16:34:52.330123 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.330094 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf"] Sep 08 16:34:52.383868 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.383843 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4fs\" (UniqueName: \"kubernetes.io/projected/19577729-a864-4ada-be14-f753ba3e785f-kube-api-access-hm4fs\") pod \"payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf\" (UID: \"19577729-a864-4ada-be14-f753ba3e785f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:34:52.383973 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.383956 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19577729-a864-4ada-be14-f753ba3e785f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf\" (UID: \"19577729-a864-4ada-be14-f753ba3e785f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:34:52.484779 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.484745 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4fs\" (UniqueName: \"kubernetes.io/projected/19577729-a864-4ada-be14-f753ba3e785f-kube-api-access-hm4fs\") pod \"payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf\" (UID: \"19577729-a864-4ada-be14-f753ba3e785f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:34:52.484948 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.484865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19577729-a864-4ada-be14-f753ba3e785f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf\" (UID: \"19577729-a864-4ada-be14-f753ba3e785f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:34:52.485523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.485491 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19577729-a864-4ada-be14-f753ba3e785f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf\" (UID: \"19577729-a864-4ada-be14-f753ba3e785f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:34:52.515476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.515444 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4fs\" (UniqueName: \"kubernetes.io/projected/19577729-a864-4ada-be14-f753ba3e785f-kube-api-access-hm4fs\") pod \"payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf\" (UID: \"19577729-a864-4ada-be14-f753ba3e785f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:34:52.606419 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.606325 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:34:52.638228 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.638197 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="507547eae4cf7052374a1950f588a0f4fb959a99a3ad8a26bdf99a462bc67611" Sep 08 16:34:52.638228 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.638209 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-140370438503216" Sep 08 16:34:52.649173 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:34:52.649141 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-140370438503216\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" podUID="bdc3faa9-e2c5-434a-a308-9482f1fc7719" pod="opendatahub/test-curl-7851-140370438503216" Sep 08 16:34:52.894215 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:52.894187 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf"] Sep 08 16:34:52.896544 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:34:52.896505 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19577729_a864_4ada_be14_f753ba3e785f.slice/crio-70066d8af32d6921ab7eb8e301e5a398b736cd65036018ae595da4176333880c WatchSource:0}: Error finding container 70066d8af32d6921ab7eb8e301e5a398b736cd65036018ae595da4176333880c: Status 404 returned error can't find the container with id 70066d8af32d6921ab7eb8e301e5a398b736cd65036018ae595da4176333880c Sep 08 16:34:53.052266 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:34:53.052229 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-140370438503216\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" podUID="bdc3faa9-e2c5-434a-a308-9482f1fc7719" pod="opendatahub/test-curl-7851-140370438503216" Sep 08 16:34:53.053393 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:53.053373 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdc3faa9-e2c5-434a-a308-9482f1fc7719" path="/var/lib/kubelet/pods/bdc3faa9-e2c5-434a-a308-9482f1fc7719/volumes" Sep 08 16:34:53.644940 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:53.644906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" event={"ID":"19577729-a864-4ada-be14-f753ba3e785f","Type":"ContainerStarted","Data":"230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775"} Sep 08 16:34:53.645170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:53.644962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" event={"ID":"19577729-a864-4ada-be14-f753ba3e785f","Type":"ContainerStarted","Data":"70066d8af32d6921ab7eb8e301e5a398b736cd65036018ae595da4176333880c"} Sep 08 16:34:53.645170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:34:53.645087 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:35:01.831002 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.830947 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" podStartSLOduration=9.830928737 podStartE2EDuration="9.830928737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:53.666838184 +0000 UTC m=+721.137824641" watchObservedRunningTime="2026-09-08 16:35:01.830928737 +0000 UTC m=+729.301915192" Sep 08 16:35:01.831498 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.831091 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9"] Sep 08 16:35:01.835753 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.835733 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:01.842476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.842125 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7553b8c9\"" Sep 08 16:35:01.842476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.842165 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7553b8c9-dockercfg-bh2gq\"" Sep 08 16:35:01.862154 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.862117 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9"] Sep 08 16:35:01.863657 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.863629 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e506d0-3403-461a-9f98-50f6fbcf96b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9\" (UID: \"66e506d0-3403-461a-9f98-50f6fbcf96b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:01.863798 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.863702 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ftq\" (UniqueName: \"kubernetes.io/projected/66e506d0-3403-461a-9f98-50f6fbcf96b6-kube-api-access-85ftq\") pod \"payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9\" (UID: \"66e506d0-3403-461a-9f98-50f6fbcf96b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:01.964868 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.964828 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e506d0-3403-461a-9f98-50f6fbcf96b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9\" (UID: \"66e506d0-3403-461a-9f98-50f6fbcf96b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:01.965098 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.964896 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ftq\" (UniqueName: \"kubernetes.io/projected/66e506d0-3403-461a-9f98-50f6fbcf96b6-kube-api-access-85ftq\") pod \"payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9\" (UID: \"66e506d0-3403-461a-9f98-50f6fbcf96b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:01.965654 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.965633 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e506d0-3403-461a-9f98-50f6fbcf96b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9\" (UID: \"66e506d0-3403-461a-9f98-50f6fbcf96b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:01.976482 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:01.976449 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ftq\" (UniqueName: \"kubernetes.io/projected/66e506d0-3403-461a-9f98-50f6fbcf96b6-kube-api-access-85ftq\") pod \"payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9\" (UID: \"66e506d0-3403-461a-9f98-50f6fbcf96b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:02.119974 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.119891 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz"] Sep 08 16:35:02.123674 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.123656 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:02.137509 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.137482 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz"] Sep 08 16:35:02.146505 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.146476 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:02.166840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.166812 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgv7\" (UniqueName: \"kubernetes.io/projected/6b4ce4e1-5c58-4b2a-b322-662fe702a487-kube-api-access-sqgv7\") pod \"payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz\" (UID: \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\") " pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:02.167005 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.166867 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4ce4e1-5c58-4b2a-b322-662fe702a487-plugins-config-volume\") pod \"payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz\" (UID: \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\") " pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:02.267706 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.267673 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgv7\" (UniqueName: \"kubernetes.io/projected/6b4ce4e1-5c58-4b2a-b322-662fe702a487-kube-api-access-sqgv7\") pod \"payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz\" (UID: \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\") " pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:02.267887 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.267764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4ce4e1-5c58-4b2a-b322-662fe702a487-plugins-config-volume\") pod \"payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz\" (UID: \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\") " pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:02.268610 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.268576 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4ce4e1-5c58-4b2a-b322-662fe702a487-plugins-config-volume\") pod \"payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz\" (UID: \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\") " pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:02.284248 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.284213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgv7\" (UniqueName: \"kubernetes.io/projected/6b4ce4e1-5c58-4b2a-b322-662fe702a487-kube-api-access-sqgv7\") pod \"payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz\" (UID: \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\") " pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:02.427517 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.427487 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9"] Sep 08 16:35:02.429331 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:35:02.429306 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e506d0_3403_461a_9f98_50f6fbcf96b6.slice/crio-0b170e1f511d4a471e982f9729d8b0c2e9a8956dedcbbdcd6dbeb71eb2ef3b5e WatchSource:0}: Error finding container 0b170e1f511d4a471e982f9729d8b0c2e9a8956dedcbbdcd6dbeb71eb2ef3b5e: Status 404 returned error can't find the container with id 0b170e1f511d4a471e982f9729d8b0c2e9a8956dedcbbdcd6dbeb71eb2ef3b5e Sep 08 16:35:02.434450 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.434432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:02.693491 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.693394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" event={"ID":"66e506d0-3403-461a-9f98-50f6fbcf96b6","Type":"ContainerStarted","Data":"d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8"} Sep 08 16:35:02.693491 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.693462 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" event={"ID":"66e506d0-3403-461a-9f98-50f6fbcf96b6","Type":"ContainerStarted","Data":"0b170e1f511d4a471e982f9729d8b0c2e9a8956dedcbbdcd6dbeb71eb2ef3b5e"} Sep 08 16:35:02.693794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.693575 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:02.711867 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.711801 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" podStartSLOduration=1.711779425 podStartE2EDuration="1.711779425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:02.71028182 +0000 UTC m=+730.181268302" watchObservedRunningTime="2026-09-08 16:35:02.711779425 +0000 UTC m=+730.182765910" Sep 08 16:35:02.716672 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.716639 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz"] Sep 08 16:35:02.721894 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:35:02.721866 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4ce4e1_5c58_4b2a_b322_662fe702a487.slice/crio-9a2a57d31c86941584b5beb8e445a0a48b058c088c27c5b30b7691e023995e3f WatchSource:0}: Error finding container 9a2a57d31c86941584b5beb8e445a0a48b058c088c27c5b30b7691e023995e3f: Status 404 returned error can't find the container with id 9a2a57d31c86941584b5beb8e445a0a48b058c088c27c5b30b7691e023995e3f Sep 08 16:35:02.999561 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:02.999467 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k"] Sep 08 16:35:03.003840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.003817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:03.006636 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.006610 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-616972\"" Sep 08 16:35:03.007678 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.007652 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4"] Sep 08 16:35:03.011887 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.011864 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:03.019268 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.019236 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k"] Sep 08 16:35:03.024403 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.024378 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4"] Sep 08 16:35:03.075288 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.075245 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1503393-41df-488e-bec4-657fbe5044d1-plugins-config-volume\") pod \"payload-processing-e2e-autores-616972-7454b64d6c-5whn4\" (UID: \"a1503393-41df-488e-bec4-657fbe5044d1\") " pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:03.075464 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.075332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-kube-api-access-wlzbj\") pod \"payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k\" (UID: \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:03.075464 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.075423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k\" (UID: \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:03.075542 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.075472 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5np5\" (UniqueName: \"kubernetes.io/projected/a1503393-41df-488e-bec4-657fbe5044d1-kube-api-access-p5np5\") pod \"payload-processing-e2e-autores-616972-7454b64d6c-5whn4\" (UID: \"a1503393-41df-488e-bec4-657fbe5044d1\") " pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:03.176044 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.175982 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-kube-api-access-wlzbj\") pod \"payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k\" (UID: \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:03.176252 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.176053 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k\" (UID: \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:03.176252 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.176096 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5np5\" (UniqueName: \"kubernetes.io/projected/a1503393-41df-488e-bec4-657fbe5044d1-kube-api-access-p5np5\") pod \"payload-processing-e2e-autores-616972-7454b64d6c-5whn4\" (UID: \"a1503393-41df-488e-bec4-657fbe5044d1\") " pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:03.176252 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.176154 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1503393-41df-488e-bec4-657fbe5044d1-plugins-config-volume\") pod \"payload-processing-e2e-autores-616972-7454b64d6c-5whn4\" (UID: \"a1503393-41df-488e-bec4-657fbe5044d1\") " pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:03.176936 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.176906 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k\" (UID: \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:03.177092 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.176962 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1503393-41df-488e-bec4-657fbe5044d1-plugins-config-volume\") pod \"payload-processing-e2e-autores-616972-7454b64d6c-5whn4\" (UID: \"a1503393-41df-488e-bec4-657fbe5044d1\") " pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:03.185743 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.185707 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5np5\" (UniqueName: \"kubernetes.io/projected/a1503393-41df-488e-bec4-657fbe5044d1-kube-api-access-p5np5\") pod \"payload-processing-e2e-autores-616972-7454b64d6c-5whn4\" (UID: \"a1503393-41df-488e-bec4-657fbe5044d1\") " pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:03.186293 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.186266 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-kube-api-access-wlzbj\") pod \"payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k\" (UID: \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:03.320898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.320797 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:03.334356 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.334313 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:03.608089 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.608060 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k"] Sep 08 16:35:03.610407 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:35:03.610380 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944d811c_95de_4ed6_9247_8e4d2ebb4f1c.slice/crio-9e4830d2c3f9099c9ab2d044c2e6edd62ed14f4f29d25f30d091ce3b70cdb597 WatchSource:0}: Error finding container 9e4830d2c3f9099c9ab2d044c2e6edd62ed14f4f29d25f30d091ce3b70cdb597: Status 404 returned error can't find the container with id 9e4830d2c3f9099c9ab2d044c2e6edd62ed14f4f29d25f30d091ce3b70cdb597 Sep 08 16:35:03.638659 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.638630 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4"] Sep 08 16:35:03.640332 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:35:03.640304 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1503393_41df_488e_bec4_657fbe5044d1.slice/crio-45887a4aa4144c9b0c2d95c37bddb386508bf77e9d52c0b725919760944c3c6f WatchSource:0}: Error finding container 45887a4aa4144c9b0c2d95c37bddb386508bf77e9d52c0b725919760944c3c6f: Status 404 returned error can't find the container with id 45887a4aa4144c9b0c2d95c37bddb386508bf77e9d52c0b725919760944c3c6f Sep 08 16:35:03.703242 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.702528 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" event={"ID":"6b4ce4e1-5c58-4b2a-b322-662fe702a487","Type":"ContainerStarted","Data":"48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653"} Sep 08 16:35:03.703242 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.702598 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" event={"ID":"6b4ce4e1-5c58-4b2a-b322-662fe702a487","Type":"ContainerStarted","Data":"9a2a57d31c86941584b5beb8e445a0a48b058c088c27c5b30b7691e023995e3f"} Sep 08 16:35:03.705078 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.705049 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" event={"ID":"a1503393-41df-488e-bec4-657fbe5044d1","Type":"ContainerStarted","Data":"45887a4aa4144c9b0c2d95c37bddb386508bf77e9d52c0b725919760944c3c6f"} Sep 08 16:35:03.706963 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.706934 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" event={"ID":"944d811c-95de-4ed6-9247-8e4d2ebb4f1c","Type":"ContainerStarted","Data":"a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81"} Sep 08 16:35:03.707109 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.706984 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" event={"ID":"944d811c-95de-4ed6-9247-8e4d2ebb4f1c","Type":"ContainerStarted","Data":"9e4830d2c3f9099c9ab2d044c2e6edd62ed14f4f29d25f30d091ce3b70cdb597"} Sep 08 16:35:03.707177 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.707114 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:03.727555 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.727505 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" podStartSLOduration=1.727483946 podStartE2EDuration="1.727483946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:03.726726221 +0000 UTC m=+731.197712678" watchObservedRunningTime="2026-09-08 16:35:03.727483946 +0000 UTC m=+731.198470401" Sep 08 16:35:03.746482 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:03.746430 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" podStartSLOduration=1.746414448 podStartE2EDuration="1.746414448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:03.745814988 +0000 UTC m=+731.216801446" watchObservedRunningTime="2026-09-08 16:35:03.746414448 +0000 UTC m=+731.217400904" Sep 08 16:35:04.651566 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:04.651534 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:35:04.715099 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:04.715008 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" event={"ID":"a1503393-41df-488e-bec4-657fbe5044d1","Type":"ContainerStarted","Data":"828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde"} Sep 08 16:35:04.715558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:04.715539 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:04.757228 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:04.757176 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" podStartSLOduration=2.757159388 podStartE2EDuration="2.757159388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:04.753267626 +0000 UTC m=+732.224254084" watchObservedRunningTime="2026-09-08 16:35:04.757159388 +0000 UTC m=+732.228145845" Sep 08 16:35:05.721503 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:05.721464 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:13.709667 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:13.709634 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:14.717149 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:14.717115 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:35:15.723853 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:15.723823 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:16.728490 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:16.728462 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:35:21.008767 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.008732 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m"] Sep 08 16:35:21.012634 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.012610 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:21.017875 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.017851 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-870dc0\"" Sep 08 16:35:21.018011 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.017892 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-870dc0-dockercfg-58wfp\"" Sep 08 16:35:21.026948 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.026917 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m"] Sep 08 16:35:21.117512 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.117476 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb"] Sep 08 16:35:21.121174 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.121148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:21.137855 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.137834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb"] Sep 08 16:35:21.162687 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.162640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lqj\" (UniqueName: \"kubernetes.io/projected/73328598-385a-4df6-a835-212033ffd056-kube-api-access-k2lqj\") pod \"payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m\" (UID: \"73328598-385a-4df6-a835-212033ffd056\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:21.163079 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.162879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73328598-385a-4df6-a835-212033ffd056-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m\" (UID: \"73328598-385a-4df6-a835-212033ffd056\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:21.264154 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.264056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73328598-385a-4df6-a835-212033ffd056-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m\" (UID: \"73328598-385a-4df6-a835-212033ffd056\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:21.264337 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.264194 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lqj\" (UniqueName: \"kubernetes.io/projected/73328598-385a-4df6-a835-212033ffd056-kube-api-access-k2lqj\") pod \"payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m\" (UID: \"73328598-385a-4df6-a835-212033ffd056\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:21.264337 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.264245 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54d10d-0859-4fad-8077-ed62494a4632-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb\" (UID: \"bb54d10d-0859-4fad-8077-ed62494a4632\") " pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:21.264337 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.264303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfn4t\" (UniqueName: \"kubernetes.io/projected/bb54d10d-0859-4fad-8077-ed62494a4632-kube-api-access-dfn4t\") pod \"payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb\" (UID: \"bb54d10d-0859-4fad-8077-ed62494a4632\") " pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:21.264831 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.264805 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73328598-385a-4df6-a835-212033ffd056-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m\" (UID: \"73328598-385a-4df6-a835-212033ffd056\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:21.274496 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.274473 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lqj\" (UniqueName: \"kubernetes.io/projected/73328598-385a-4df6-a835-212033ffd056-kube-api-access-k2lqj\") pod \"payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m\" (UID: \"73328598-385a-4df6-a835-212033ffd056\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:21.327049 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.326997 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:21.365590 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.365545 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54d10d-0859-4fad-8077-ed62494a4632-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb\" (UID: \"bb54d10d-0859-4fad-8077-ed62494a4632\") " pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:21.365778 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.365602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfn4t\" (UniqueName: \"kubernetes.io/projected/bb54d10d-0859-4fad-8077-ed62494a4632-kube-api-access-dfn4t\") pod \"payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb\" (UID: \"bb54d10d-0859-4fad-8077-ed62494a4632\") " pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:21.366310 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.366285 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54d10d-0859-4fad-8077-ed62494a4632-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb\" (UID: \"bb54d10d-0859-4fad-8077-ed62494a4632\") " pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:21.375100 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.375071 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfn4t\" (UniqueName: \"kubernetes.io/projected/bb54d10d-0859-4fad-8077-ed62494a4632-kube-api-access-dfn4t\") pod \"payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb\" (UID: \"bb54d10d-0859-4fad-8077-ed62494a4632\") " pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:21.432130 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.432093 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:21.610162 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.610063 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m"] Sep 08 16:35:21.613378 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:35:21.613352 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73328598_385a_4df6_a835_212033ffd056.slice/crio-32aa9515b25f988f77464e2b0d897f1753a9d22bd6e0cc1faf41a9ae8b221b8a WatchSource:0}: Error finding container 32aa9515b25f988f77464e2b0d897f1753a9d22bd6e0cc1faf41a9ae8b221b8a: Status 404 returned error can't find the container with id 32aa9515b25f988f77464e2b0d897f1753a9d22bd6e0cc1faf41a9ae8b221b8a Sep 08 16:35:21.711717 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.711692 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb"] Sep 08 16:35:21.714304 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:35:21.714269 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb54d10d_0859_4fad_8077_ed62494a4632.slice/crio-6768aef25e8ba2e83a9ff622db3ca932637e39f9b92dcb3a44b2a8a13056c312 WatchSource:0}: Error finding container 6768aef25e8ba2e83a9ff622db3ca932637e39f9b92dcb3a44b2a8a13056c312: Status 404 returned error can't find the container with id 6768aef25e8ba2e83a9ff622db3ca932637e39f9b92dcb3a44b2a8a13056c312 Sep 08 16:35:21.813904 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.813873 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" event={"ID":"bb54d10d-0859-4fad-8077-ed62494a4632","Type":"ContainerStarted","Data":"c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d"} Sep 08 16:35:21.814114 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.813927 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" event={"ID":"bb54d10d-0859-4fad-8077-ed62494a4632","Type":"ContainerStarted","Data":"6768aef25e8ba2e83a9ff622db3ca932637e39f9b92dcb3a44b2a8a13056c312"} Sep 08 16:35:21.814114 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.814001 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:21.816000 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.815968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" event={"ID":"73328598-385a-4df6-a835-212033ffd056","Type":"ContainerStarted","Data":"e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4"} Sep 08 16:35:21.816148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.816045 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:21.816148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.816072 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" event={"ID":"73328598-385a-4df6-a835-212033ffd056","Type":"ContainerStarted","Data":"32aa9515b25f988f77464e2b0d897f1753a9d22bd6e0cc1faf41a9ae8b221b8a"} Sep 08 16:35:21.842710 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.842651 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" podStartSLOduration=0.842634882 podStartE2EDuration="842.634882ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:21.840893095 +0000 UTC m=+749.311879553" watchObservedRunningTime="2026-09-08 16:35:21.842634882 +0000 UTC m=+749.313621338" Sep 08 16:35:21.874817 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:21.874708 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" podStartSLOduration=1.874690419 podStartE2EDuration="1.874690419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:21.873481663 +0000 UTC m=+749.344468118" watchObservedRunningTime="2026-09-08 16:35:21.874690419 +0000 UTC m=+749.345676874" Sep 08 16:35:32.823612 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:32.823581 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:35:32.824158 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:32.823699 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:35:43.053852 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.053819 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz"] Sep 08 16:35:43.054350 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.054089 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" podUID="6b4ce4e1-5c58-4b2a-b322-662fe702a487" containerName="payload-processing" containerID="cri-o://48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653" gracePeriod=30 Sep 08 16:35:43.070117 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.070092 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9"] Sep 08 16:35:43.070385 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.070280 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" podUID="66e506d0-3403-461a-9f98-50f6fbcf96b6" containerName="payload-pre-processing" containerID="cri-o://d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8" gracePeriod=30 Sep 08 16:35:43.342829 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.342803 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:43.345700 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.345679 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:43.381341 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.381315 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqgv7\" (UniqueName: \"kubernetes.io/projected/6b4ce4e1-5c58-4b2a-b322-662fe702a487-kube-api-access-sqgv7\") pod \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\" (UID: \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\") " Sep 08 16:35:43.381479 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.381406 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85ftq\" (UniqueName: \"kubernetes.io/projected/66e506d0-3403-461a-9f98-50f6fbcf96b6-kube-api-access-85ftq\") pod \"66e506d0-3403-461a-9f98-50f6fbcf96b6\" (UID: \"66e506d0-3403-461a-9f98-50f6fbcf96b6\") " Sep 08 16:35:43.381479 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.381448 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e506d0-3403-461a-9f98-50f6fbcf96b6-plugins-config-volume\") pod \"66e506d0-3403-461a-9f98-50f6fbcf96b6\" (UID: \"66e506d0-3403-461a-9f98-50f6fbcf96b6\") " Sep 08 16:35:43.381574 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.381483 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4ce4e1-5c58-4b2a-b322-662fe702a487-plugins-config-volume\") pod \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\" (UID: \"6b4ce4e1-5c58-4b2a-b322-662fe702a487\") " Sep 08 16:35:43.381868 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.381843 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4ce4e1-5c58-4b2a-b322-662fe702a487-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b4ce4e1-5c58-4b2a-b322-662fe702a487" (UID: "6b4ce4e1-5c58-4b2a-b322-662fe702a487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:35:43.381868 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.381849 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e506d0-3403-461a-9f98-50f6fbcf96b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66e506d0-3403-461a-9f98-50f6fbcf96b6" (UID: "66e506d0-3403-461a-9f98-50f6fbcf96b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:35:43.383695 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.383674 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4ce4e1-5c58-4b2a-b322-662fe702a487-kube-api-access-sqgv7" (OuterVolumeSpecName: "kube-api-access-sqgv7") pod "6b4ce4e1-5c58-4b2a-b322-662fe702a487" (UID: "6b4ce4e1-5c58-4b2a-b322-662fe702a487"). InnerVolumeSpecName "kube-api-access-sqgv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:35:43.383840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.383728 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e506d0-3403-461a-9f98-50f6fbcf96b6-kube-api-access-85ftq" (OuterVolumeSpecName: "kube-api-access-85ftq") pod "66e506d0-3403-461a-9f98-50f6fbcf96b6" (UID: "66e506d0-3403-461a-9f98-50f6fbcf96b6"). InnerVolumeSpecName "kube-api-access-85ftq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:35:43.482606 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.482575 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqgv7\" (UniqueName: \"kubernetes.io/projected/6b4ce4e1-5c58-4b2a-b322-662fe702a487-kube-api-access-sqgv7\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:35:43.482606 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.482602 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85ftq\" (UniqueName: \"kubernetes.io/projected/66e506d0-3403-461a-9f98-50f6fbcf96b6-kube-api-access-85ftq\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:35:43.482606 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.482611 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e506d0-3403-461a-9f98-50f6fbcf96b6-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:35:43.482829 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.482619 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4ce4e1-5c58-4b2a-b322-662fe702a487-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:35:43.931151 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.931115 2567 generic.go:356] "Generic (PLEG): container finished" podID="66e506d0-3403-461a-9f98-50f6fbcf96b6" containerID="d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8" exitCode=0 Sep 08 16:35:43.931334 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.931163 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" Sep 08 16:35:43.931334 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.931200 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" event={"ID":"66e506d0-3403-461a-9f98-50f6fbcf96b6","Type":"ContainerDied","Data":"d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8"} Sep 08 16:35:43.931334 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.931307 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9" event={"ID":"66e506d0-3403-461a-9f98-50f6fbcf96b6","Type":"ContainerDied","Data":"0b170e1f511d4a471e982f9729d8b0c2e9a8956dedcbbdcd6dbeb71eb2ef3b5e"} Sep 08 16:35:43.931485 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.931336 2567 scope.go:117] "RemoveContainer" containerID="d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8" Sep 08 16:35:43.932621 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.932567 2567 generic.go:356] "Generic (PLEG): container finished" podID="6b4ce4e1-5c58-4b2a-b322-662fe702a487" containerID="48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653" exitCode=0 Sep 08 16:35:43.932708 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.932620 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" Sep 08 16:35:43.932708 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.932653 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" event={"ID":"6b4ce4e1-5c58-4b2a-b322-662fe702a487","Type":"ContainerDied","Data":"48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653"} Sep 08 16:35:43.932809 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.932707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz" event={"ID":"6b4ce4e1-5c58-4b2a-b322-662fe702a487","Type":"ContainerDied","Data":"9a2a57d31c86941584b5beb8e445a0a48b058c088c27c5b30b7691e023995e3f"} Sep 08 16:35:43.940717 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.940705 2567 scope.go:117] "RemoveContainer" containerID="d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8" Sep 08 16:35:43.941007 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:35:43.940978 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8\": container with ID starting with d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8 not found: ID does not exist" containerID="d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8" Sep 08 16:35:43.941149 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.941045 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8"} err="failed to get container status \"d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8\": rpc error: code = NotFound desc = could not find container \"d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8\": container with ID starting with d29b075f33f3f4f8bd5208b6db201c103a5d0086ec4dcf147af84fb927ad69b8 not found: ID does not exist" Sep 08 16:35:43.941149 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.941070 2567 scope.go:117] "RemoveContainer" containerID="48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653" Sep 08 16:35:43.948974 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.948956 2567 scope.go:117] "RemoveContainer" containerID="48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653" Sep 08 16:35:43.949248 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:35:43.949231 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653\": container with ID starting with 48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653 not found: ID does not exist" containerID="48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653" Sep 08 16:35:43.949307 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.949255 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653"} err="failed to get container status \"48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653\": rpc error: code = NotFound desc = could not find container \"48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653\": container with ID starting with 48e2a39ff6db71a3fdd6ed979298e241c81ae10b9a18880289a88cb109998653 not found: ID does not exist" Sep 08 16:35:43.993721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:43.993677 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz"] Sep 08 16:35:44.007560 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:44.007520 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7553b8c9-6f756897b7-wggbz"] Sep 08 16:35:44.023506 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:44.023471 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9"] Sep 08 16:35:44.035330 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:44.035300 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7553b8c9-564cb5bfd6-m6hj9"] Sep 08 16:35:45.053404 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:45.053370 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e506d0-3403-461a-9f98-50f6fbcf96b6" path="/var/lib/kubelet/pods/66e506d0-3403-461a-9f98-50f6fbcf96b6/volumes" Sep 08 16:35:45.053781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:45.053739 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b4ce4e1-5c58-4b2a-b322-662fe702a487" path="/var/lib/kubelet/pods/6b4ce4e1-5c58-4b2a-b322-662fe702a487/volumes" Sep 08 16:35:47.157691 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.157658 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc"] Sep 08 16:35:47.158163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.158148 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b4ce4e1-5c58-4b2a-b322-662fe702a487" containerName="payload-processing" Sep 08 16:35:47.158218 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.158166 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b4ce4e1-5c58-4b2a-b322-662fe702a487" containerName="payload-processing" Sep 08 16:35:47.158218 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.158190 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e506d0-3403-461a-9f98-50f6fbcf96b6" containerName="payload-pre-processing" Sep 08 16:35:47.158218 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.158195 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e506d0-3403-461a-9f98-50f6fbcf96b6" containerName="payload-pre-processing" Sep 08 16:35:47.158314 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.158257 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b4ce4e1-5c58-4b2a-b322-662fe702a487" containerName="payload-processing" Sep 08 16:35:47.158314 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.158270 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e506d0-3403-461a-9f98-50f6fbcf96b6" containerName="payload-pre-processing" Sep 08 16:35:47.162761 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.162740 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.165771 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.165753 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-04ecbe44-kserve-self-signed-certs\"" Sep 08 16:35:47.178156 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.178126 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc"] Sep 08 16:35:47.220771 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.220737 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-model-cache\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.220936 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.220782 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tls-certs\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.220936 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.220812 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp7dw\" (UniqueName: \"kubernetes.io/projected/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-kube-api-access-bp7dw\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.220936 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.220849 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tmp-dir\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.221135 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.220945 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-dshm\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.221191 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.221173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-home\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.322361 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.322331 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-home\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.322571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.322383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-model-cache\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.322571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.322403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tls-certs\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.322571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.322418 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7dw\" (UniqueName: \"kubernetes.io/projected/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-kube-api-access-bp7dw\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.322571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.322542 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tmp-dir\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.322799 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.322606 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-dshm\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.322857 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.322803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-model-cache\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.322857 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.322803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-home\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.322942 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.322893 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tmp-dir\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.324861 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.324836 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-dshm\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.325170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.325151 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tls-certs\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.342575 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.342544 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7dw\" (UniqueName: \"kubernetes.io/projected/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-kube-api-access-bp7dw\") pod \"e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.473646 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.473561 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:47.776161 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.776102 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc"] Sep 08 16:35:47.778429 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:35:47.778405 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96acfdcb_5eaf_4568_a0a8_2fcfcc2a9c72.slice/crio-d96c2b5ea9d99dd41d2e844ee5e3f19b01b2a8e00ef3127125648aacd88039bd WatchSource:0}: Error finding container d96c2b5ea9d99dd41d2e844ee5e3f19b01b2a8e00ef3127125648aacd88039bd: Status 404 returned error can't find the container with id d96c2b5ea9d99dd41d2e844ee5e3f19b01b2a8e00ef3127125648aacd88039bd Sep 08 16:35:47.957894 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.957856 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" event={"ID":"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72","Type":"ContainerStarted","Data":"973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589"} Sep 08 16:35:47.958123 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.957908 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" event={"ID":"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72","Type":"ContainerStarted","Data":"d96c2b5ea9d99dd41d2e844ee5e3f19b01b2a8e00ef3127125648aacd88039bd"} Sep 08 16:35:47.990967 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:47.990919 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" podStartSLOduration=0.990904241 podStartE2EDuration="990.904241ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:47.989455064 +0000 UTC m=+775.460441520" watchObservedRunningTime="2026-09-08 16:35:47.990904241 +0000 UTC m=+775.461890697" Sep 08 16:35:48.961288 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:48.961250 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg"] Sep 08 16:35:48.965581 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:48.965561 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:35:48.969090 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:48.969012 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-421f5bb0-dockercfg-5gcpn\"" Sep 08 16:35:48.969438 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:48.969418 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-421f5bb0\"" Sep 08 16:35:48.982600 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:48.982574 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg"] Sep 08 16:35:49.038602 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.038559 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ac51b0-e579-4148-9e8a-7e013b535ea5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg\" (UID: \"59ac51b0-e579-4148-9e8a-7e013b535ea5\") " pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:35:49.038785 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.038725 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbdz\" (UniqueName: \"kubernetes.io/projected/59ac51b0-e579-4148-9e8a-7e013b535ea5-kube-api-access-rdbdz\") pod \"payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg\" (UID: \"59ac51b0-e579-4148-9e8a-7e013b535ea5\") " pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:35:49.139916 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.139885 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbdz\" (UniqueName: \"kubernetes.io/projected/59ac51b0-e579-4148-9e8a-7e013b535ea5-kube-api-access-rdbdz\") pod \"payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg\" (UID: \"59ac51b0-e579-4148-9e8a-7e013b535ea5\") " pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:35:49.140128 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.139977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ac51b0-e579-4148-9e8a-7e013b535ea5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg\" (UID: \"59ac51b0-e579-4148-9e8a-7e013b535ea5\") " pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:35:49.140666 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.140645 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ac51b0-e579-4148-9e8a-7e013b535ea5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg\" (UID: \"59ac51b0-e579-4148-9e8a-7e013b535ea5\") " pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:35:49.150620 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.150596 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbdz\" (UniqueName: \"kubernetes.io/projected/59ac51b0-e579-4148-9e8a-7e013b535ea5-kube-api-access-rdbdz\") pod \"payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg\" (UID: \"59ac51b0-e579-4148-9e8a-7e013b535ea5\") " pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:35:49.276729 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.276627 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:35:49.550669 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.550595 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg"] Sep 08 16:35:49.553891 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:35:49.553864 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ac51b0_e579_4148_9e8a_7e013b535ea5.slice/crio-3df85a3711e2a08b5153ad312845814f50101ed38b1ee600207b945bca77f4b6 WatchSource:0}: Error finding container 3df85a3711e2a08b5153ad312845814f50101ed38b1ee600207b945bca77f4b6: Status 404 returned error can't find the container with id 3df85a3711e2a08b5153ad312845814f50101ed38b1ee600207b945bca77f4b6 Sep 08 16:35:49.970830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.970790 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" event={"ID":"59ac51b0-e579-4148-9e8a-7e013b535ea5","Type":"ContainerStarted","Data":"41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961"} Sep 08 16:35:49.971237 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.970842 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" event={"ID":"59ac51b0-e579-4148-9e8a-7e013b535ea5","Type":"ContainerStarted","Data":"3df85a3711e2a08b5153ad312845814f50101ed38b1ee600207b945bca77f4b6"} Sep 08 16:35:49.971237 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:49.970872 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:35:50.003874 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:50.003821 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" podStartSLOduration=2.003807049 podStartE2EDuration="2.003807049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:50.003274761 +0000 UTC m=+777.474261217" watchObservedRunningTime="2026-09-08 16:35:50.003807049 +0000 UTC m=+777.474793523" Sep 08 16:35:57.474510 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:57.474469 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:57.474906 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:57.474538 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:57.486724 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:57.486704 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:35:58.025916 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:35:58.025885 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:36:00.978053 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:00.978005 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:36:18.809907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:18.809723 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg"] Sep 08 16:36:18.810465 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:18.810255 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" podUID="59ac51b0-e579-4148-9e8a-7e013b535ea5" containerName="payload-processing" containerID="cri-o://41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961" gracePeriod=30 Sep 08 16:36:19.063037 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.062961 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:36:19.123802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.123771 2567 generic.go:356] "Generic (PLEG): container finished" podID="59ac51b0-e579-4148-9e8a-7e013b535ea5" containerID="41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961" exitCode=0 Sep 08 16:36:19.123966 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.123823 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" Sep 08 16:36:19.123966 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.123864 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" event={"ID":"59ac51b0-e579-4148-9e8a-7e013b535ea5","Type":"ContainerDied","Data":"41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961"} Sep 08 16:36:19.123966 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.123922 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg" event={"ID":"59ac51b0-e579-4148-9e8a-7e013b535ea5","Type":"ContainerDied","Data":"3df85a3711e2a08b5153ad312845814f50101ed38b1ee600207b945bca77f4b6"} Sep 08 16:36:19.123966 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.123952 2567 scope.go:117] "RemoveContainer" containerID="41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961" Sep 08 16:36:19.127292 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.127271 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdbdz\" (UniqueName: \"kubernetes.io/projected/59ac51b0-e579-4148-9e8a-7e013b535ea5-kube-api-access-rdbdz\") pod \"59ac51b0-e579-4148-9e8a-7e013b535ea5\" (UID: \"59ac51b0-e579-4148-9e8a-7e013b535ea5\") " Sep 08 16:36:19.127435 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.127337 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ac51b0-e579-4148-9e8a-7e013b535ea5-plugins-config-volume\") pod \"59ac51b0-e579-4148-9e8a-7e013b535ea5\" (UID: \"59ac51b0-e579-4148-9e8a-7e013b535ea5\") " Sep 08 16:36:19.127697 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.127670 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ac51b0-e579-4148-9e8a-7e013b535ea5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59ac51b0-e579-4148-9e8a-7e013b535ea5" (UID: "59ac51b0-e579-4148-9e8a-7e013b535ea5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:36:19.129620 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.129588 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ac51b0-e579-4148-9e8a-7e013b535ea5-kube-api-access-rdbdz" (OuterVolumeSpecName: "kube-api-access-rdbdz") pod "59ac51b0-e579-4148-9e8a-7e013b535ea5" (UID: "59ac51b0-e579-4148-9e8a-7e013b535ea5"). InnerVolumeSpecName "kube-api-access-rdbdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:36:19.132868 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.132848 2567 scope.go:117] "RemoveContainer" containerID="41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961" Sep 08 16:36:19.133185 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:36:19.133167 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961\": container with ID starting with 41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961 not found: ID does not exist" containerID="41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961" Sep 08 16:36:19.133238 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.133193 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961"} err="failed to get container status \"41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961\": rpc error: code = NotFound desc = could not find container \"41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961\": container with ID starting with 41667685c472c5e464e33db2247433c8945145d6780255998613fc3d659cc961 not found: ID does not exist" Sep 08 16:36:19.228641 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.228609 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdbdz\" (UniqueName: \"kubernetes.io/projected/59ac51b0-e579-4148-9e8a-7e013b535ea5-kube-api-access-rdbdz\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:19.228641 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.228634 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ac51b0-e579-4148-9e8a-7e013b535ea5-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:19.446029 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.445998 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg"] Sep 08 16:36:19.452081 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:19.452060 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-421f5bb0-598bf4ff98-mpxpg"] Sep 08 16:36:21.054269 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:21.054238 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ac51b0-e579-4148-9e8a-7e013b535ea5" path="/var/lib/kubelet/pods/59ac51b0-e579-4148-9e8a-7e013b535ea5/volumes" Sep 08 16:36:23.873260 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:23.873224 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc"] Sep 08 16:36:24.577785 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.577751 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l"] Sep 08 16:36:24.578201 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.578188 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ac51b0-e579-4148-9e8a-7e013b535ea5" containerName="payload-processing" Sep 08 16:36:24.578256 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.578203 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ac51b0-e579-4148-9e8a-7e013b535ea5" containerName="payload-processing" Sep 08 16:36:24.578295 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.578289 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ac51b0-e579-4148-9e8a-7e013b535ea5" containerName="payload-processing" Sep 08 16:36:24.582568 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.582553 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:24.587295 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.587271 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6dd85108-dockercfg-nqbzh\"" Sep 08 16:36:24.587484 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.587331 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6dd85108\"" Sep 08 16:36:24.609685 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.609656 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l"] Sep 08 16:36:24.681810 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.681773 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b55a489-14f3-486b-bbf8-85160429d792-plugins-config-volume\") pod \"payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l\" (UID: \"7b55a489-14f3-486b-bbf8-85160429d792\") " pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:24.681981 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.681843 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdl9\" (UniqueName: \"kubernetes.io/projected/7b55a489-14f3-486b-bbf8-85160429d792-kube-api-access-kzdl9\") pod \"payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l\" (UID: \"7b55a489-14f3-486b-bbf8-85160429d792\") " pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:24.782627 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.782590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b55a489-14f3-486b-bbf8-85160429d792-plugins-config-volume\") pod \"payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l\" (UID: \"7b55a489-14f3-486b-bbf8-85160429d792\") " pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:24.782794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.782670 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdl9\" (UniqueName: \"kubernetes.io/projected/7b55a489-14f3-486b-bbf8-85160429d792-kube-api-access-kzdl9\") pod \"payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l\" (UID: \"7b55a489-14f3-486b-bbf8-85160429d792\") " pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:24.783336 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.783318 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b55a489-14f3-486b-bbf8-85160429d792-plugins-config-volume\") pod \"payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l\" (UID: \"7b55a489-14f3-486b-bbf8-85160429d792\") " pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:24.791647 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.791623 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdl9\" (UniqueName: \"kubernetes.io/projected/7b55a489-14f3-486b-bbf8-85160429d792-kube-api-access-kzdl9\") pod \"payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l\" (UID: \"7b55a489-14f3-486b-bbf8-85160429d792\") " pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:24.893511 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:24.893423 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:25.179771 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:25.179741 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l"] Sep 08 16:36:25.181036 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:36:25.180996 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b55a489_14f3_486b_bbf8_85160429d792.slice/crio-fc060a4e5ce2235fb78c8e538b0fd44c5dfa6c0b9da76a98e00c6453f5fc9bf1 WatchSource:0}: Error finding container fc060a4e5ce2235fb78c8e538b0fd44c5dfa6c0b9da76a98e00c6453f5fc9bf1: Status 404 returned error can't find the container with id fc060a4e5ce2235fb78c8e538b0fd44c5dfa6c0b9da76a98e00c6453f5fc9bf1 Sep 08 16:36:26.165677 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:26.165638 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" event={"ID":"7b55a489-14f3-486b-bbf8-85160429d792","Type":"ContainerStarted","Data":"c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f"} Sep 08 16:36:26.166122 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:26.165696 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" event={"ID":"7b55a489-14f3-486b-bbf8-85160429d792","Type":"ContainerStarted","Data":"fc060a4e5ce2235fb78c8e538b0fd44c5dfa6c0b9da76a98e00c6453f5fc9bf1"} Sep 08 16:36:26.166122 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:26.165790 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:26.184725 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:26.184663 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" podStartSLOduration=2.184646287 podStartE2EDuration="2.184646287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:36:26.183743136 +0000 UTC m=+813.654729617" watchObservedRunningTime="2026-09-08 16:36:26.184646287 +0000 UTC m=+813.655632743" Sep 08 16:36:37.173945 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:37.173914 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:36:38.897980 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:38.897920 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" podUID="96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" containerName="main" containerID="cri-o://973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589" gracePeriod=45 Sep 08 16:36:39.152012 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.151951 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:36:39.228610 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.228577 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-home\") pod \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " Sep 08 16:36:39.228786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.228619 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-dshm\") pod \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " Sep 08 16:36:39.228786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.228671 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tls-certs\") pod \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " Sep 08 16:36:39.228786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.228703 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tmp-dir\") pod \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " Sep 08 16:36:39.228786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.228764 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp7dw\" (UniqueName: \"kubernetes.io/projected/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-kube-api-access-bp7dw\") pod \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " Sep 08 16:36:39.228998 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.228813 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-model-cache\") pod \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\" (UID: \"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72\") " Sep 08 16:36:39.228998 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.228896 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-home" (OuterVolumeSpecName: "home") pod "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" (UID: "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:36:39.229133 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.229068 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" (UID: "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:36:39.229259 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.229240 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:39.229319 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.229264 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:39.229319 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.229243 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-model-cache" (OuterVolumeSpecName: "model-cache") pod "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" (UID: "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:36:39.231176 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.231149 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" (UID: "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:36:39.231301 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.231241 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-dshm" (OuterVolumeSpecName: "dshm") pod "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" (UID: "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:36:39.231301 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.231257 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-kube-api-access-bp7dw" (OuterVolumeSpecName: "kube-api-access-bp7dw") pod "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" (UID: "96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72"). InnerVolumeSpecName "kube-api-access-bp7dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:36:39.235627 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.235599 2567 generic.go:356] "Generic (PLEG): container finished" podID="96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" containerID="973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589" exitCode=0 Sep 08 16:36:39.235718 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.235641 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" Sep 08 16:36:39.235718 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.235678 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" event={"ID":"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72","Type":"ContainerDied","Data":"973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589"} Sep 08 16:36:39.235789 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.235723 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc" event={"ID":"96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72","Type":"ContainerDied","Data":"d96c2b5ea9d99dd41d2e844ee5e3f19b01b2a8e00ef3127125648aacd88039bd"} Sep 08 16:36:39.235789 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.235745 2567 scope.go:117] "RemoveContainer" containerID="973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589" Sep 08 16:36:39.251046 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.251006 2567 scope.go:117] "RemoveContainer" containerID="973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589" Sep 08 16:36:39.251306 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:36:39.251288 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589\": container with ID starting with 973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589 not found: ID does not exist" containerID="973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589" Sep 08 16:36:39.251362 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.251315 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589"} err="failed to get container status \"973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589\": rpc error: code = NotFound desc = could not find container \"973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589\": container with ID starting with 973ae232fc8d7f006a94575a7f342688e54d09c6e5b513fda4a1de9061154589 not found: ID does not exist" Sep 08 16:36:39.272035 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.271997 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc"] Sep 08 16:36:39.278764 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.278740 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-04ecbe44-kserve-778ddfb7b-5x6tc"] Sep 08 16:36:39.336931 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.331178 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp7dw\" (UniqueName: \"kubernetes.io/projected/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-kube-api-access-bp7dw\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:39.336931 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.331217 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:39.336931 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.331241 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:39.336931 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:39.331254 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:41.053725 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:41.053689 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" path="/var/lib/kubelet/pods/96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72/volumes" Sep 08 16:36:42.836731 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:42.836683 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4"] Sep 08 16:36:42.837202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:42.837001 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" podUID="a1503393-41df-488e-bec4-657fbe5044d1" containerName="payload-processing" containerID="cri-o://828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde" gracePeriod=30 Sep 08 16:36:42.846055 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:42.846004 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k"] Sep 08 16:36:42.846267 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:42.846222 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" podUID="944d811c-95de-4ed6-9247-8e4d2ebb4f1c" containerName="payload-pre-processing" containerID="cri-o://a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81" gracePeriod=30 Sep 08 16:36:43.125151 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.125129 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:36:43.128868 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.128849 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:36:43.261058 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.261008 2567 generic.go:356] "Generic (PLEG): container finished" podID="a1503393-41df-488e-bec4-657fbe5044d1" containerID="828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde" exitCode=0 Sep 08 16:36:43.261246 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.261074 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" Sep 08 16:36:43.261246 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.261117 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" event={"ID":"a1503393-41df-488e-bec4-657fbe5044d1","Type":"ContainerDied","Data":"828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde"} Sep 08 16:36:43.261246 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.261174 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4" event={"ID":"a1503393-41df-488e-bec4-657fbe5044d1","Type":"ContainerDied","Data":"45887a4aa4144c9b0c2d95c37bddb386508bf77e9d52c0b725919760944c3c6f"} Sep 08 16:36:43.261246 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.261198 2567 scope.go:117] "RemoveContainer" containerID="828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde" Sep 08 16:36:43.262436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.262419 2567 generic.go:356] "Generic (PLEG): container finished" podID="944d811c-95de-4ed6-9247-8e4d2ebb4f1c" containerID="a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81" exitCode=0 Sep 08 16:36:43.262529 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.262460 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" Sep 08 16:36:43.262529 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.262472 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" event={"ID":"944d811c-95de-4ed6-9247-8e4d2ebb4f1c","Type":"ContainerDied","Data":"a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81"} Sep 08 16:36:43.262529 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.262503 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k" event={"ID":"944d811c-95de-4ed6-9247-8e4d2ebb4f1c","Type":"ContainerDied","Data":"9e4830d2c3f9099c9ab2d044c2e6edd62ed14f4f29d25f30d091ce3b70cdb597"} Sep 08 16:36:43.269830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.269816 2567 scope.go:117] "RemoveContainer" containerID="828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde" Sep 08 16:36:43.270135 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:36:43.270118 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde\": container with ID starting with 828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde not found: ID does not exist" containerID="828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde" Sep 08 16:36:43.270199 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.270142 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde"} err="failed to get container status \"828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde\": rpc error: code = NotFound desc = could not find container \"828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde\": container with ID starting with 828a75f823162b1d52c2094bc500f7765588ea372f35dd6352585a2a97be0dde not found: ID does not exist" Sep 08 16:36:43.270199 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.270158 2567 scope.go:117] "RemoveContainer" containerID="a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81" Sep 08 16:36:43.276342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.276324 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1503393-41df-488e-bec4-657fbe5044d1-plugins-config-volume\") pod \"a1503393-41df-488e-bec4-657fbe5044d1\" (UID: \"a1503393-41df-488e-bec4-657fbe5044d1\") " Sep 08 16:36:43.276443 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.276398 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-plugins-config-volume\") pod \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\" (UID: \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\") " Sep 08 16:36:43.276485 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.276444 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-kube-api-access-wlzbj\") pod \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\" (UID: \"944d811c-95de-4ed6-9247-8e4d2ebb4f1c\") " Sep 08 16:36:43.276485 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.276480 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5np5\" (UniqueName: \"kubernetes.io/projected/a1503393-41df-488e-bec4-657fbe5044d1-kube-api-access-p5np5\") pod \"a1503393-41df-488e-bec4-657fbe5044d1\" (UID: \"a1503393-41df-488e-bec4-657fbe5044d1\") " Sep 08 16:36:43.276764 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.276737 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1503393-41df-488e-bec4-657fbe5044d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1503393-41df-488e-bec4-657fbe5044d1" (UID: "a1503393-41df-488e-bec4-657fbe5044d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:36:43.276841 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.276788 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "944d811c-95de-4ed6-9247-8e4d2ebb4f1c" (UID: "944d811c-95de-4ed6-9247-8e4d2ebb4f1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:36:43.278110 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.278090 2567 scope.go:117] "RemoveContainer" containerID="a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81" Sep 08 16:36:43.278436 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:36:43.278412 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81\": container with ID starting with a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81 not found: ID does not exist" containerID="a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81" Sep 08 16:36:43.278534 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.278439 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81"} err="failed to get container status \"a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81\": rpc error: code = NotFound desc = could not find container \"a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81\": container with ID starting with a33c983eea2e014c997b9897bc32ada4d0ec9664644ae280f96c773766610c81 not found: ID does not exist" Sep 08 16:36:43.278633 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.278617 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1503393-41df-488e-bec4-657fbe5044d1-kube-api-access-p5np5" (OuterVolumeSpecName: "kube-api-access-p5np5") pod "a1503393-41df-488e-bec4-657fbe5044d1" (UID: "a1503393-41df-488e-bec4-657fbe5044d1"). InnerVolumeSpecName "kube-api-access-p5np5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:36:43.278829 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.278808 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-kube-api-access-wlzbj" (OuterVolumeSpecName: "kube-api-access-wlzbj") pod "944d811c-95de-4ed6-9247-8e4d2ebb4f1c" (UID: "944d811c-95de-4ed6-9247-8e4d2ebb4f1c"). InnerVolumeSpecName "kube-api-access-wlzbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:36:43.377844 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.377768 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-kube-api-access-wlzbj\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:43.377844 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.377795 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5np5\" (UniqueName: \"kubernetes.io/projected/a1503393-41df-488e-bec4-657fbe5044d1-kube-api-access-p5np5\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:43.377844 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.377805 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1503393-41df-488e-bec4-657fbe5044d1-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:43.377844 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.377814 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944d811c-95de-4ed6-9247-8e4d2ebb4f1c-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:43.589554 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.589527 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k"] Sep 08 16:36:43.597591 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.597565 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-616972-7fc6c8588b-vjj6k"] Sep 08 16:36:43.617693 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.617668 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4"] Sep 08 16:36:43.621073 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:43.621049 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-616972-7454b64d6c-5whn4"] Sep 08 16:36:45.053277 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:45.053246 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="944d811c-95de-4ed6-9247-8e4d2ebb4f1c" path="/var/lib/kubelet/pods/944d811c-95de-4ed6-9247-8e4d2ebb4f1c/volumes" Sep 08 16:36:45.053638 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:45.053604 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1503393-41df-488e-bec4-657fbe5044d1" path="/var/lib/kubelet/pods/a1503393-41df-488e-bec4-657fbe5044d1/volumes" Sep 08 16:36:53.519056 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519001 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2"] Sep 08 16:36:53.519664 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519643 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1503393-41df-488e-bec4-657fbe5044d1" containerName="payload-processing" Sep 08 16:36:53.519664 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519663 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1503393-41df-488e-bec4-657fbe5044d1" containerName="payload-processing" Sep 08 16:36:53.519830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519685 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" containerName="main" Sep 08 16:36:53.519830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519690 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" containerName="main" Sep 08 16:36:53.519830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519702 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="944d811c-95de-4ed6-9247-8e4d2ebb4f1c" containerName="payload-pre-processing" Sep 08 16:36:53.519830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519708 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="944d811c-95de-4ed6-9247-8e4d2ebb4f1c" containerName="payload-pre-processing" Sep 08 16:36:53.519830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519774 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="944d811c-95de-4ed6-9247-8e4d2ebb4f1c" containerName="payload-pre-processing" Sep 08 16:36:53.519830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519782 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1503393-41df-488e-bec4-657fbe5044d1" containerName="payload-processing" Sep 08 16:36:53.519830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.519789 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="96acfdcb-5eaf-4568-a0a8-2fcfcc2a9c72" containerName="main" Sep 08 16:36:53.524528 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.524511 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" Sep 08 16:36:53.526831 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.526809 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzslb\"" Sep 08 16:36:53.526960 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.526811 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:36:53.530652 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.530630 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2"] Sep 08 16:36:53.575074 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.575010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90e6d5e-47df-4ecd-b20a-0d493b1d6488-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2\" (UID: \"f90e6d5e-47df-4ecd-b20a-0d493b1d6488\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" Sep 08 16:36:53.676006 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.675970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90e6d5e-47df-4ecd-b20a-0d493b1d6488-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2\" (UID: \"f90e6d5e-47df-4ecd-b20a-0d493b1d6488\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" Sep 08 16:36:53.676598 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.676577 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90e6d5e-47df-4ecd-b20a-0d493b1d6488-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2\" (UID: \"f90e6d5e-47df-4ecd-b20a-0d493b1d6488\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" Sep 08 16:36:53.835754 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:53.835661 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" Sep 08 16:36:54.106684 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:54.106603 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2"] Sep 08 16:36:54.110001 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:36:54.109966 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90e6d5e_47df_4ecd_b20a_0d493b1d6488.slice/crio-ca4712f2846090589f52118347daa103f069ccd0e58dd9dbde16b444de90029f WatchSource:0}: Error finding container ca4712f2846090589f52118347daa103f069ccd0e58dd9dbde16b444de90029f: Status 404 returned error can't find the container with id ca4712f2846090589f52118347daa103f069ccd0e58dd9dbde16b444de90029f Sep 08 16:36:54.321913 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:54.321878 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" event={"ID":"f90e6d5e-47df-4ecd-b20a-0d493b1d6488","Type":"ContainerStarted","Data":"ca4712f2846090589f52118347daa103f069ccd0e58dd9dbde16b444de90029f"} Sep 08 16:36:57.341200 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.341155 2567 generic.go:356] "Generic (PLEG): container finished" podID="f90e6d5e-47df-4ecd-b20a-0d493b1d6488" containerID="045aa947b99d6c374285935ba1879c0bae9ac52f4761502547f8892e1261dff6" exitCode=0 Sep 08 16:36:57.341665 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.341242 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" event={"ID":"f90e6d5e-47df-4ecd-b20a-0d493b1d6488","Type":"ContainerDied","Data":"045aa947b99d6c374285935ba1879c0bae9ac52f4761502547f8892e1261dff6"} Sep 08 16:36:57.465984 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.465908 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k"] Sep 08 16:36:57.469489 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.469474 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:36:57.472643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.472624 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-384f4f-dockercfg-c5szv\"" Sep 08 16:36:57.473715 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.473699 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-384f4f\"" Sep 08 16:36:57.496883 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.496857 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k"] Sep 08 16:36:57.517507 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.517474 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-plugins-config-volume\") pod \"payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k\" (UID: \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\") " pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:36:57.517663 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.517643 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vf82\" (UniqueName: \"kubernetes.io/projected/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-kube-api-access-5vf82\") pod \"payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k\" (UID: \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\") " pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:36:57.531819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.531796 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v"] Sep 08 16:36:57.535537 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.535523 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:36:57.580914 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.580887 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v"] Sep 08 16:36:57.618845 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.618814 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vf82\" (UniqueName: \"kubernetes.io/projected/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-kube-api-access-5vf82\") pod \"payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k\" (UID: \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\") " pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:36:57.619049 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.618855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-plugins-config-volume\") pod \"payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k\" (UID: \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\") " pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:36:57.619049 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.618934 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7673a92-62d0-451d-9c2d-4d1717d83ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v\" (UID: \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:36:57.619148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.619062 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5rx8\" (UniqueName: \"kubernetes.io/projected/a7673a92-62d0-451d-9c2d-4d1717d83ef1-kube-api-access-d5rx8\") pod \"payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v\" (UID: \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:36:57.619614 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.619595 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-plugins-config-volume\") pod \"payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k\" (UID: \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\") " pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:36:57.629747 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.629725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vf82\" (UniqueName: \"kubernetes.io/projected/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-kube-api-access-5vf82\") pod \"payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k\" (UID: \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\") " pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:36:57.719884 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.719798 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7673a92-62d0-451d-9c2d-4d1717d83ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v\" (UID: \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:36:57.719884 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.719846 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5rx8\" (UniqueName: \"kubernetes.io/projected/a7673a92-62d0-451d-9c2d-4d1717d83ef1-kube-api-access-d5rx8\") pod \"payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v\" (UID: \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:36:57.720539 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.720516 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7673a92-62d0-451d-9c2d-4d1717d83ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v\" (UID: \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:36:57.733952 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.733917 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5rx8\" (UniqueName: \"kubernetes.io/projected/a7673a92-62d0-451d-9c2d-4d1717d83ef1-kube-api-access-d5rx8\") pod \"payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v\" (UID: \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:36:57.779657 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.779629 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:36:57.846455 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:57.846388 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:36:58.062339 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.062192 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k"] Sep 08 16:36:58.065115 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:36:58.065090 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f1cc30_a121_47c7_a7c6_f7b40a2d5247.slice/crio-c35f5bf615b8fe3e48490aed3eb080d6251e454cd43d74f0dfd1e1215b213a02 WatchSource:0}: Error finding container c35f5bf615b8fe3e48490aed3eb080d6251e454cd43d74f0dfd1e1215b213a02: Status 404 returned error can't find the container with id c35f5bf615b8fe3e48490aed3eb080d6251e454cd43d74f0dfd1e1215b213a02 Sep 08 16:36:58.154203 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.154169 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v"] Sep 08 16:36:58.157518 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:36:58.157489 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7673a92_62d0_451d_9c2d_4d1717d83ef1.slice/crio-91a2898350885b5e54228c482503b379fc08078b760efc8979e878431da03d1a WatchSource:0}: Error finding container 91a2898350885b5e54228c482503b379fc08078b760efc8979e878431da03d1a: Status 404 returned error can't find the container with id 91a2898350885b5e54228c482503b379fc08078b760efc8979e878431da03d1a Sep 08 16:36:58.348872 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.348776 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" event={"ID":"a7673a92-62d0-451d-9c2d-4d1717d83ef1","Type":"ContainerStarted","Data":"a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6"} Sep 08 16:36:58.348872 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.348853 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:36:58.349389 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.348876 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" event={"ID":"a7673a92-62d0-451d-9c2d-4d1717d83ef1","Type":"ContainerStarted","Data":"91a2898350885b5e54228c482503b379fc08078b760efc8979e878431da03d1a"} Sep 08 16:36:58.350583 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.350551 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" event={"ID":"86f1cc30-a121-47c7-a7c6-f7b40a2d5247","Type":"ContainerStarted","Data":"f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f"} Sep 08 16:36:58.350708 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.350598 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" event={"ID":"86f1cc30-a121-47c7-a7c6-f7b40a2d5247","Type":"ContainerStarted","Data":"c35f5bf615b8fe3e48490aed3eb080d6251e454cd43d74f0dfd1e1215b213a02"} Sep 08 16:36:58.404718 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.404654 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" podStartSLOduration=1.404632337 podStartE2EDuration="1.404632337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:36:58.40268827 +0000 UTC m=+845.873674729" watchObservedRunningTime="2026-09-08 16:36:58.404632337 +0000 UTC m=+845.875618794" Sep 08 16:36:58.475943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.475915 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" Sep 08 16:36:58.481377 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.481336 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" podStartSLOduration=1.481321072 podStartE2EDuration="1.481321072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:36:58.47926152 +0000 UTC m=+845.950248009" watchObservedRunningTime="2026-09-08 16:36:58.481321072 +0000 UTC m=+845.952307565" Sep 08 16:36:58.528838 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.528804 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90e6d5e-47df-4ecd-b20a-0d493b1d6488-maas-api-service-ca\") pod \"f90e6d5e-47df-4ecd-b20a-0d493b1d6488\" (UID: \"f90e6d5e-47df-4ecd-b20a-0d493b1d6488\") " Sep 08 16:36:58.529181 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.529159 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90e6d5e-47df-4ecd-b20a-0d493b1d6488-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f90e6d5e-47df-4ecd-b20a-0d493b1d6488" (UID: "f90e6d5e-47df-4ecd-b20a-0d493b1d6488"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:36:58.629703 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:58.629619 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90e6d5e-47df-4ecd-b20a-0d493b1d6488-maas-api-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:36:59.357629 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:59.357541 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" Sep 08 16:36:59.357629 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:59.357588 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2" event={"ID":"f90e6d5e-47df-4ecd-b20a-0d493b1d6488","Type":"ContainerDied","Data":"ca4712f2846090589f52118347daa103f069ccd0e58dd9dbde16b444de90029f"} Sep 08 16:36:59.358180 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:59.357636 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca4712f2846090589f52118347daa103f069ccd0e58dd9dbde16b444de90029f" Sep 08 16:36:59.358180 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:36:59.357798 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:37:04.638697 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:04.638660 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l"] Sep 08 16:37:04.639126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:04.638950 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" podUID="7b55a489-14f3-486b-bbf8-85160429d792" containerName="payload-processing" containerID="cri-o://c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f" gracePeriod=30 Sep 08 16:37:04.809774 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:04.809736 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2"] Sep 08 16:37:04.812122 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:04.812091 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6dd85108-kvvg2"] Sep 08 16:37:04.901685 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:04.901624 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:37:04.992193 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:04.992148 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b55a489-14f3-486b-bbf8-85160429d792-plugins-config-volume\") pod \"7b55a489-14f3-486b-bbf8-85160429d792\" (UID: \"7b55a489-14f3-486b-bbf8-85160429d792\") " Sep 08 16:37:04.992373 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:04.992217 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzdl9\" (UniqueName: \"kubernetes.io/projected/7b55a489-14f3-486b-bbf8-85160429d792-kube-api-access-kzdl9\") pod \"7b55a489-14f3-486b-bbf8-85160429d792\" (UID: \"7b55a489-14f3-486b-bbf8-85160429d792\") " Sep 08 16:37:04.992538 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:04.992515 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b55a489-14f3-486b-bbf8-85160429d792-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b55a489-14f3-486b-bbf8-85160429d792" (UID: "7b55a489-14f3-486b-bbf8-85160429d792"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:37:04.994509 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:04.994484 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b55a489-14f3-486b-bbf8-85160429d792-kube-api-access-kzdl9" (OuterVolumeSpecName: "kube-api-access-kzdl9") pod "7b55a489-14f3-486b-bbf8-85160429d792" (UID: "7b55a489-14f3-486b-bbf8-85160429d792"). InnerVolumeSpecName "kube-api-access-kzdl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:37:05.053083 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.053047 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90e6d5e-47df-4ecd-b20a-0d493b1d6488" path="/var/lib/kubelet/pods/f90e6d5e-47df-4ecd-b20a-0d493b1d6488/volumes" Sep 08 16:37:05.093410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.093369 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b55a489-14f3-486b-bbf8-85160429d792-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:37:05.093410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.093406 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzdl9\" (UniqueName: \"kubernetes.io/projected/7b55a489-14f3-486b-bbf8-85160429d792-kube-api-access-kzdl9\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:37:05.390205 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.390169 2567 generic.go:356] "Generic (PLEG): container finished" podID="7b55a489-14f3-486b-bbf8-85160429d792" containerID="c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f" exitCode=0 Sep 08 16:37:05.390406 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.390217 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" Sep 08 16:37:05.390406 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.390255 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" event={"ID":"7b55a489-14f3-486b-bbf8-85160429d792","Type":"ContainerDied","Data":"c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f"} Sep 08 16:37:05.390406 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.390314 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l" event={"ID":"7b55a489-14f3-486b-bbf8-85160429d792","Type":"ContainerDied","Data":"fc060a4e5ce2235fb78c8e538b0fd44c5dfa6c0b9da76a98e00c6453f5fc9bf1"} Sep 08 16:37:05.390406 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.390345 2567 scope.go:117] "RemoveContainer" containerID="c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f" Sep 08 16:37:05.399680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.399664 2567 scope.go:117] "RemoveContainer" containerID="c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f" Sep 08 16:37:05.399969 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:37:05.399938 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f\": container with ID starting with c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f not found: ID does not exist" containerID="c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f" Sep 08 16:37:05.400059 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.399969 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f"} err="failed to get container status \"c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f\": rpc error: code = NotFound desc = could not find container \"c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f\": container with ID starting with c2e07637757bbc02839e739e640322953b194ceada9e5662e0738b1e0a01c16f not found: ID does not exist" Sep 08 16:37:05.408387 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.408362 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l"] Sep 08 16:37:05.413089 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:05.413065 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6dd85108-f88f45cd7-9jj8l"] Sep 08 16:37:07.053240 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:07.053210 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b55a489-14f3-486b-bbf8-85160429d792" path="/var/lib/kubelet/pods/7b55a489-14f3-486b-bbf8-85160429d792/volumes" Sep 08 16:37:09.360472 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:09.360435 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:37:10.365326 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.365295 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:37:10.965081 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.965046 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx"] Sep 08 16:37:10.965561 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.965543 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90e6d5e-47df-4ecd-b20a-0d493b1d6488" containerName="revoke-keys" Sep 08 16:37:10.965654 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.965564 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90e6d5e-47df-4ecd-b20a-0d493b1d6488" containerName="revoke-keys" Sep 08 16:37:10.965654 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.965577 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b55a489-14f3-486b-bbf8-85160429d792" containerName="payload-processing" Sep 08 16:37:10.965654 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.965584 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b55a489-14f3-486b-bbf8-85160429d792" containerName="payload-processing" Sep 08 16:37:10.965804 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.965682 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b55a489-14f3-486b-bbf8-85160429d792" containerName="payload-processing" Sep 08 16:37:10.965804 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.965695 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90e6d5e-47df-4ecd-b20a-0d493b1d6488" containerName="revoke-keys" Sep 08 16:37:10.970353 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.970333 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:10.973085 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.973066 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2fb1f7b7-dockercfg-srnk2\"" Sep 08 16:37:10.973269 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.973253 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2fb1f7b7\"" Sep 08 16:37:10.981835 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:10.981812 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx"] Sep 08 16:37:11.045528 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.045492 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh"] Sep 08 16:37:11.049133 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.049112 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:11.051673 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.051652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnsv5\" (UniqueName: \"kubernetes.io/projected/651e65d6-ea68-45ac-b055-d6b64d761232-kube-api-access-qnsv5\") pod \"payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx\" (UID: \"651e65d6-ea68-45ac-b055-d6b64d761232\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:11.051759 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.051685 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651e65d6-ea68-45ac-b055-d6b64d761232-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx\" (UID: \"651e65d6-ea68-45ac-b055-d6b64d761232\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:11.061287 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.061266 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh"] Sep 08 16:37:11.152993 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.152954 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2715c5d3-3570-4fa1-896d-61d024c678ac-plugins-config-volume\") pod \"payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh\" (UID: \"2715c5d3-3570-4fa1-896d-61d024c678ac\") " pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:11.153200 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.153006 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstld\" (UniqueName: \"kubernetes.io/projected/2715c5d3-3570-4fa1-896d-61d024c678ac-kube-api-access-vstld\") pod \"payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh\" (UID: \"2715c5d3-3570-4fa1-896d-61d024c678ac\") " pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:11.153200 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.153110 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnsv5\" (UniqueName: \"kubernetes.io/projected/651e65d6-ea68-45ac-b055-d6b64d761232-kube-api-access-qnsv5\") pod \"payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx\" (UID: \"651e65d6-ea68-45ac-b055-d6b64d761232\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:11.153200 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.153155 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651e65d6-ea68-45ac-b055-d6b64d761232-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx\" (UID: \"651e65d6-ea68-45ac-b055-d6b64d761232\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:11.153833 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.153813 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651e65d6-ea68-45ac-b055-d6b64d761232-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx\" (UID: \"651e65d6-ea68-45ac-b055-d6b64d761232\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:11.168844 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.168815 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnsv5\" (UniqueName: \"kubernetes.io/projected/651e65d6-ea68-45ac-b055-d6b64d761232-kube-api-access-qnsv5\") pod \"payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx\" (UID: \"651e65d6-ea68-45ac-b055-d6b64d761232\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:11.253792 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.253700 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2715c5d3-3570-4fa1-896d-61d024c678ac-plugins-config-volume\") pod \"payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh\" (UID: \"2715c5d3-3570-4fa1-896d-61d024c678ac\") " pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:11.253792 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.253751 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstld\" (UniqueName: \"kubernetes.io/projected/2715c5d3-3570-4fa1-896d-61d024c678ac-kube-api-access-vstld\") pod \"payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh\" (UID: \"2715c5d3-3570-4fa1-896d-61d024c678ac\") " pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:11.254472 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.254446 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2715c5d3-3570-4fa1-896d-61d024c678ac-plugins-config-volume\") pod \"payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh\" (UID: \"2715c5d3-3570-4fa1-896d-61d024c678ac\") " pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:11.262807 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.262782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstld\" (UniqueName: \"kubernetes.io/projected/2715c5d3-3570-4fa1-896d-61d024c678ac-kube-api-access-vstld\") pod \"payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh\" (UID: \"2715c5d3-3570-4fa1-896d-61d024c678ac\") " pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:11.281106 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.281081 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:11.359946 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.359912 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:11.562646 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.562623 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx"] Sep 08 16:37:11.564402 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:37:11.564374 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651e65d6_ea68_45ac_b055_d6b64d761232.slice/crio-1c81047d873cb9b812f9914e682886c411f69787f6ba87ff8c93da7dfc3858bb WatchSource:0}: Error finding container 1c81047d873cb9b812f9914e682886c411f69787f6ba87ff8c93da7dfc3858bb: Status 404 returned error can't find the container with id 1c81047d873cb9b812f9914e682886c411f69787f6ba87ff8c93da7dfc3858bb Sep 08 16:37:11.640050 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:11.640005 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh"] Sep 08 16:37:11.642206 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:37:11.642179 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2715c5d3_3570_4fa1_896d_61d024c678ac.slice/crio-0162d46c5e8b986c65193054c621759b181c55afedfcc291962c85f0607b6e72 WatchSource:0}: Error finding container 0162d46c5e8b986c65193054c621759b181c55afedfcc291962c85f0607b6e72: Status 404 returned error can't find the container with id 0162d46c5e8b986c65193054c621759b181c55afedfcc291962c85f0607b6e72 Sep 08 16:37:12.435455 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:12.435419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" event={"ID":"651e65d6-ea68-45ac-b055-d6b64d761232","Type":"ContainerStarted","Data":"dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162"} Sep 08 16:37:12.435662 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:12.435485 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:12.435662 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:12.435509 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" event={"ID":"651e65d6-ea68-45ac-b055-d6b64d761232","Type":"ContainerStarted","Data":"1c81047d873cb9b812f9914e682886c411f69787f6ba87ff8c93da7dfc3858bb"} Sep 08 16:37:12.437090 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:12.437061 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" event={"ID":"2715c5d3-3570-4fa1-896d-61d024c678ac","Type":"ContainerStarted","Data":"e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1"} Sep 08 16:37:12.437194 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:12.437112 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" event={"ID":"2715c5d3-3570-4fa1-896d-61d024c678ac","Type":"ContainerStarted","Data":"0162d46c5e8b986c65193054c621759b181c55afedfcc291962c85f0607b6e72"} Sep 08 16:37:12.437194 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:12.437158 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:12.466683 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:12.466623 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" podStartSLOduration=2.466608158 podStartE2EDuration="2.466608158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:37:12.464218931 +0000 UTC m=+859.935205387" watchObservedRunningTime="2026-09-08 16:37:12.466608158 +0000 UTC m=+859.937594613" Sep 08 16:37:12.495988 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:12.495939 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" podStartSLOduration=1.495923641 podStartE2EDuration="1.495923641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:37:12.494523558 +0000 UTC m=+859.965510030" watchObservedRunningTime="2026-09-08 16:37:12.495923641 +0000 UTC m=+859.966910096" Sep 08 16:37:23.444090 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:23.444059 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:23.444589 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:23.444140 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:33.735169 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:33.735135 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb"] Sep 08 16:37:33.743714 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:33.743684 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" Sep 08 16:37:33.746363 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:33.746170 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:37:33.746363 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:33.746200 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzslb\"" Sep 08 16:37:33.746680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:33.746657 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb"] Sep 08 16:37:33.765487 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:33.765458 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13e9c44-db8a-4ac7-a245-6e87fe7b1065-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb\" (UID: \"f13e9c44-db8a-4ac7-a245-6e87fe7b1065\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" Sep 08 16:37:33.866946 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:33.866916 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13e9c44-db8a-4ac7-a245-6e87fe7b1065-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb\" (UID: \"f13e9c44-db8a-4ac7-a245-6e87fe7b1065\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" Sep 08 16:37:33.867560 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:33.867537 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13e9c44-db8a-4ac7-a245-6e87fe7b1065-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb\" (UID: \"f13e9c44-db8a-4ac7-a245-6e87fe7b1065\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" Sep 08 16:37:34.056125 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:34.056032 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" Sep 08 16:37:34.332984 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:34.332950 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb"] Sep 08 16:37:34.333743 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:37:34.333710 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13e9c44_db8a_4ac7_a245_6e87fe7b1065.slice/crio-c15c171c26a2cc179fab475004544fbe6f207e86de27fa3aedf3e85f78eb5528 WatchSource:0}: Error finding container c15c171c26a2cc179fab475004544fbe6f207e86de27fa3aedf3e85f78eb5528: Status 404 returned error can't find the container with id c15c171c26a2cc179fab475004544fbe6f207e86de27fa3aedf3e85f78eb5528 Sep 08 16:37:34.556855 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:34.556822 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" event={"ID":"f13e9c44-db8a-4ac7-a245-6e87fe7b1065","Type":"ContainerStarted","Data":"c15c171c26a2cc179fab475004544fbe6f207e86de27fa3aedf3e85f78eb5528"} Sep 08 16:37:35.563247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:35.563219 2567 generic.go:356] "Generic (PLEG): container finished" podID="f13e9c44-db8a-4ac7-a245-6e87fe7b1065" containerID="be3f21790998890dbf0805ecf6e6f2ac773eb5971917ed368f2980147208c1e6" exitCode=0 Sep 08 16:37:35.563610 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:35.563305 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" event={"ID":"f13e9c44-db8a-4ac7-a245-6e87fe7b1065","Type":"ContainerDied","Data":"be3f21790998890dbf0805ecf6e6f2ac773eb5971917ed368f2980147208c1e6"} Sep 08 16:37:36.717013 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:36.716988 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" Sep 08 16:37:36.795763 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:36.795727 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13e9c44-db8a-4ac7-a245-6e87fe7b1065-maas-api-service-ca\") pod \"f13e9c44-db8a-4ac7-a245-6e87fe7b1065\" (UID: \"f13e9c44-db8a-4ac7-a245-6e87fe7b1065\") " Sep 08 16:37:36.796157 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:36.796125 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13e9c44-db8a-4ac7-a245-6e87fe7b1065-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f13e9c44-db8a-4ac7-a245-6e87fe7b1065" (UID: "f13e9c44-db8a-4ac7-a245-6e87fe7b1065"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:37:36.896735 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:36.896654 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13e9c44-db8a-4ac7-a245-6e87fe7b1065-maas-api-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:37:37.576794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:37.576765 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" Sep 08 16:37:37.576936 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:37.576759 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb" event={"ID":"f13e9c44-db8a-4ac7-a245-6e87fe7b1065","Type":"ContainerDied","Data":"c15c171c26a2cc179fab475004544fbe6f207e86de27fa3aedf3e85f78eb5528"} Sep 08 16:37:37.576936 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:37.576895 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c15c171c26a2cc179fab475004544fbe6f207e86de27fa3aedf3e85f78eb5528" Sep 08 16:37:40.930899 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:40.930862 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh"] Sep 08 16:37:40.931415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:40.931175 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" podUID="2715c5d3-3570-4fa1-896d-61d024c678ac" containerName="payload-processing" containerID="cri-o://e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1" gracePeriod=30 Sep 08 16:37:40.940674 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:40.940639 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx"] Sep 08 16:37:40.940881 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:40.940853 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" podUID="651e65d6-ea68-45ac-b055-d6b64d761232" containerName="payload-pre-processing" containerID="cri-o://dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162" gracePeriod=30 Sep 08 16:37:41.132049 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.131965 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb"] Sep 08 16:37:41.135577 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.135551 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fb1f7b7-pqjlb"] Sep 08 16:37:41.202220 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.202197 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:41.205296 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.205275 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:41.241389 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.241361 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651e65d6-ea68-45ac-b055-d6b64d761232-plugins-config-volume\") pod \"651e65d6-ea68-45ac-b055-d6b64d761232\" (UID: \"651e65d6-ea68-45ac-b055-d6b64d761232\") " Sep 08 16:37:41.241552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.241435 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2715c5d3-3570-4fa1-896d-61d024c678ac-plugins-config-volume\") pod \"2715c5d3-3570-4fa1-896d-61d024c678ac\" (UID: \"2715c5d3-3570-4fa1-896d-61d024c678ac\") " Sep 08 16:37:41.241552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.241474 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnsv5\" (UniqueName: \"kubernetes.io/projected/651e65d6-ea68-45ac-b055-d6b64d761232-kube-api-access-qnsv5\") pod \"651e65d6-ea68-45ac-b055-d6b64d761232\" (UID: \"651e65d6-ea68-45ac-b055-d6b64d761232\") " Sep 08 16:37:41.241552 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.241529 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vstld\" (UniqueName: \"kubernetes.io/projected/2715c5d3-3570-4fa1-896d-61d024c678ac-kube-api-access-vstld\") pod \"2715c5d3-3570-4fa1-896d-61d024c678ac\" (UID: \"2715c5d3-3570-4fa1-896d-61d024c678ac\") " Sep 08 16:37:41.241819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.241789 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2715c5d3-3570-4fa1-896d-61d024c678ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2715c5d3-3570-4fa1-896d-61d024c678ac" (UID: "2715c5d3-3570-4fa1-896d-61d024c678ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:37:41.241911 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.241815 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651e65d6-ea68-45ac-b055-d6b64d761232-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "651e65d6-ea68-45ac-b055-d6b64d761232" (UID: "651e65d6-ea68-45ac-b055-d6b64d761232"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:37:41.243812 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.243779 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651e65d6-ea68-45ac-b055-d6b64d761232-kube-api-access-qnsv5" (OuterVolumeSpecName: "kube-api-access-qnsv5") pod "651e65d6-ea68-45ac-b055-d6b64d761232" (UID: "651e65d6-ea68-45ac-b055-d6b64d761232"). InnerVolumeSpecName "kube-api-access-qnsv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:37:41.243911 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.243834 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2715c5d3-3570-4fa1-896d-61d024c678ac-kube-api-access-vstld" (OuterVolumeSpecName: "kube-api-access-vstld") pod "2715c5d3-3570-4fa1-896d-61d024c678ac" (UID: "2715c5d3-3570-4fa1-896d-61d024c678ac"). InnerVolumeSpecName "kube-api-access-vstld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:37:41.342389 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.342362 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2715c5d3-3570-4fa1-896d-61d024c678ac-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:37:41.342389 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.342387 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnsv5\" (UniqueName: \"kubernetes.io/projected/651e65d6-ea68-45ac-b055-d6b64d761232-kube-api-access-qnsv5\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:37:41.342389 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.342396 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vstld\" (UniqueName: \"kubernetes.io/projected/2715c5d3-3570-4fa1-896d-61d024c678ac-kube-api-access-vstld\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:37:41.342389 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.342404 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651e65d6-ea68-45ac-b055-d6b64d761232-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:37:41.599888 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.599799 2567 generic.go:356] "Generic (PLEG): container finished" podID="651e65d6-ea68-45ac-b055-d6b64d761232" containerID="dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162" exitCode=0 Sep 08 16:37:41.599888 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.599846 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" Sep 08 16:37:41.600127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.599880 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" event={"ID":"651e65d6-ea68-45ac-b055-d6b64d761232","Type":"ContainerDied","Data":"dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162"} Sep 08 16:37:41.600127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.599929 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx" event={"ID":"651e65d6-ea68-45ac-b055-d6b64d761232","Type":"ContainerDied","Data":"1c81047d873cb9b812f9914e682886c411f69787f6ba87ff8c93da7dfc3858bb"} Sep 08 16:37:41.600127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.599952 2567 scope.go:117] "RemoveContainer" containerID="dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162" Sep 08 16:37:41.601370 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.601350 2567 generic.go:356] "Generic (PLEG): container finished" podID="2715c5d3-3570-4fa1-896d-61d024c678ac" containerID="e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1" exitCode=0 Sep 08 16:37:41.601462 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.601392 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" Sep 08 16:37:41.601462 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.601393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" event={"ID":"2715c5d3-3570-4fa1-896d-61d024c678ac","Type":"ContainerDied","Data":"e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1"} Sep 08 16:37:41.601462 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.601456 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh" event={"ID":"2715c5d3-3570-4fa1-896d-61d024c678ac","Type":"ContainerDied","Data":"0162d46c5e8b986c65193054c621759b181c55afedfcc291962c85f0607b6e72"} Sep 08 16:37:41.617060 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.617041 2567 scope.go:117] "RemoveContainer" containerID="dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162" Sep 08 16:37:41.617380 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:37:41.617362 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162\": container with ID starting with dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162 not found: ID does not exist" containerID="dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162" Sep 08 16:37:41.617427 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.617387 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162"} err="failed to get container status \"dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162\": rpc error: code = NotFound desc = could not find container \"dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162\": container with ID starting with dbc12a54df6c9ad6236942f0936539695f5bb48d3a10cf0982b9466f1ac86162 not found: ID does not exist" Sep 08 16:37:41.617427 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.617404 2567 scope.go:117] "RemoveContainer" containerID="e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1" Sep 08 16:37:41.625977 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.625955 2567 scope.go:117] "RemoveContainer" containerID="e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1" Sep 08 16:37:41.626337 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:37:41.626318 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1\": container with ID starting with e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1 not found: ID does not exist" containerID="e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1" Sep 08 16:37:41.626394 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.626347 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1"} err="failed to get container status \"e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1\": rpc error: code = NotFound desc = could not find container \"e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1\": container with ID starting with e5c0304d7d7353fd2f7d22a132ba242097f02ecf71ae5b60e8cb54a4a7bebab1 not found: ID does not exist" Sep 08 16:37:41.645636 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.645611 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx"] Sep 08 16:37:41.650691 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.650667 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2fb1f7b7-bd48946c8-t5gzx"] Sep 08 16:37:41.670418 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.670386 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh"] Sep 08 16:37:41.672296 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:41.672271 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2fb1f7b7-55bc45bfd5-tk7zh"] Sep 08 16:37:43.054754 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:43.054712 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2715c5d3-3570-4fa1-896d-61d024c678ac" path="/var/lib/kubelet/pods/2715c5d3-3570-4fa1-896d-61d024c678ac/volumes" Sep 08 16:37:43.055268 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:43.055245 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="651e65d6-ea68-45ac-b055-d6b64d761232" path="/var/lib/kubelet/pods/651e65d6-ea68-45ac-b055-d6b64d761232/volumes" Sep 08 16:37:43.055593 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:43.055573 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f13e9c44-db8a-4ac7-a245-6e87fe7b1065" path="/var/lib/kubelet/pods/f13e9c44-db8a-4ac7-a245-6e87fe7b1065/volumes" Sep 08 16:37:52.992641 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:52.992610 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:37:52.993666 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:52.993645 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:37:59.635698 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.635661 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx"] Sep 08 16:37:59.636147 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.636098 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="651e65d6-ea68-45ac-b055-d6b64d761232" containerName="payload-pre-processing" Sep 08 16:37:59.636147 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.636109 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="651e65d6-ea68-45ac-b055-d6b64d761232" containerName="payload-pre-processing" Sep 08 16:37:59.636147 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.636118 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2715c5d3-3570-4fa1-896d-61d024c678ac" containerName="payload-processing" Sep 08 16:37:59.636147 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.636124 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2715c5d3-3570-4fa1-896d-61d024c678ac" containerName="payload-processing" Sep 08 16:37:59.636147 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.636140 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13e9c44-db8a-4ac7-a245-6e87fe7b1065" containerName="revoke-keys" Sep 08 16:37:59.636147 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.636146 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13e9c44-db8a-4ac7-a245-6e87fe7b1065" containerName="revoke-keys" Sep 08 16:37:59.636354 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.636203 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2715c5d3-3570-4fa1-896d-61d024c678ac" containerName="payload-processing" Sep 08 16:37:59.636354 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.636214 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="651e65d6-ea68-45ac-b055-d6b64d761232" containerName="payload-pre-processing" Sep 08 16:37:59.636354 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.636220 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f13e9c44-db8a-4ac7-a245-6e87fe7b1065" containerName="revoke-keys" Sep 08 16:37:59.640748 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.640730 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" Sep 08 16:37:59.645227 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.645209 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzslb\"" Sep 08 16:37:59.645339 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.645211 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:37:59.654762 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.654736 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx"] Sep 08 16:37:59.827835 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.827793 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef6ed3b-e41e-455c-90f2-d9dae60fbd9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx\" (UID: \"aef6ed3b-e41e-455c-90f2-d9dae60fbd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" Sep 08 16:37:59.929243 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.929206 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef6ed3b-e41e-455c-90f2-d9dae60fbd9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx\" (UID: \"aef6ed3b-e41e-455c-90f2-d9dae60fbd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" Sep 08 16:37:59.929851 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.929826 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef6ed3b-e41e-455c-90f2-d9dae60fbd9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx\" (UID: \"aef6ed3b-e41e-455c-90f2-d9dae60fbd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" Sep 08 16:37:59.951120 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:37:59.951091 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" Sep 08 16:38:00.220949 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:00.220857 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx"] Sep 08 16:38:00.223385 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:38:00.223356 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef6ed3b_e41e_455c_90f2_d9dae60fbd9e.slice/crio-5637c194bce4fa47c0d7ae75f5d86a3db771cecb51c4a2cf2102f300e66883da WatchSource:0}: Error finding container 5637c194bce4fa47c0d7ae75f5d86a3db771cecb51c4a2cf2102f300e66883da: Status 404 returned error can't find the container with id 5637c194bce4fa47c0d7ae75f5d86a3db771cecb51c4a2cf2102f300e66883da Sep 08 16:38:00.710631 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:00.710590 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" event={"ID":"aef6ed3b-e41e-455c-90f2-d9dae60fbd9e","Type":"ContainerStarted","Data":"651dd9a18c3107321261d33a3d24976fea8aa54a8ca1e2feeeb988ee117c50f2"} Sep 08 16:38:00.711009 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:00.710655 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" event={"ID":"aef6ed3b-e41e-455c-90f2-d9dae60fbd9e","Type":"ContainerStarted","Data":"5637c194bce4fa47c0d7ae75f5d86a3db771cecb51c4a2cf2102f300e66883da"} Sep 08 16:38:00.727413 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:00.727361 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" podStartSLOduration=1.727345594 podStartE2EDuration="1.727345594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:00.726218558 +0000 UTC m=+908.197205016" watchObservedRunningTime="2026-09-08 16:38:00.727345594 +0000 UTC m=+908.198332098" Sep 08 16:38:01.716800 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:01.716760 2567 generic.go:356] "Generic (PLEG): container finished" podID="aef6ed3b-e41e-455c-90f2-d9dae60fbd9e" containerID="651dd9a18c3107321261d33a3d24976fea8aa54a8ca1e2feeeb988ee117c50f2" exitCode=0 Sep 08 16:38:01.717220 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:01.716855 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" event={"ID":"aef6ed3b-e41e-455c-90f2-d9dae60fbd9e","Type":"ContainerDied","Data":"651dd9a18c3107321261d33a3d24976fea8aa54a8ca1e2feeeb988ee117c50f2"} Sep 08 16:38:02.860822 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:02.860792 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" Sep 08 16:38:03.056501 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:03.056425 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef6ed3b-e41e-455c-90f2-d9dae60fbd9e-maas-api-service-ca\") pod \"aef6ed3b-e41e-455c-90f2-d9dae60fbd9e\" (UID: \"aef6ed3b-e41e-455c-90f2-d9dae60fbd9e\") " Sep 08 16:38:03.056780 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:03.056756 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef6ed3b-e41e-455c-90f2-d9dae60fbd9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aef6ed3b-e41e-455c-90f2-d9dae60fbd9e" (UID: "aef6ed3b-e41e-455c-90f2-d9dae60fbd9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:38:03.157885 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:03.157849 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef6ed3b-e41e-455c-90f2-d9dae60fbd9e-maas-api-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:03.731934 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:03.731901 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" Sep 08 16:38:03.732148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:03.731936 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx" event={"ID":"aef6ed3b-e41e-455c-90f2-d9dae60fbd9e","Type":"ContainerDied","Data":"5637c194bce4fa47c0d7ae75f5d86a3db771cecb51c4a2cf2102f300e66883da"} Sep 08 16:38:03.732148 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:03.731988 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5637c194bce4fa47c0d7ae75f5d86a3db771cecb51c4a2cf2102f300e66883da" Sep 08 16:38:10.179828 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.179791 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k"] Sep 08 16:38:10.180293 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.180075 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" podUID="86f1cc30-a121-47c7-a7c6-f7b40a2d5247" containerName="payload-processing" containerID="cri-o://f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f" gracePeriod=30 Sep 08 16:38:10.184868 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.184833 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v"] Sep 08 16:38:10.185282 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.185228 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" podUID="a7673a92-62d0-451d-9c2d-4d1717d83ef1" containerName="payload-pre-processing" containerID="cri-o://a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6" gracePeriod=30 Sep 08 16:38:10.364785 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.364750 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx"] Sep 08 16:38:10.368043 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.367997 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-384f4f-vrwpx"] Sep 08 16:38:10.450889 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.450862 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:38:10.481767 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.481742 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:38:10.520723 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.520692 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-plugins-config-volume\") pod \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\" (UID: \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\") " Sep 08 16:38:10.520928 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.520765 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7673a92-62d0-451d-9c2d-4d1717d83ef1-plugins-config-volume\") pod \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\" (UID: \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\") " Sep 08 16:38:10.520928 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.520900 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5rx8\" (UniqueName: \"kubernetes.io/projected/a7673a92-62d0-451d-9c2d-4d1717d83ef1-kube-api-access-d5rx8\") pod \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\" (UID: \"a7673a92-62d0-451d-9c2d-4d1717d83ef1\") " Sep 08 16:38:10.521116 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.520962 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vf82\" (UniqueName: \"kubernetes.io/projected/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-kube-api-access-5vf82\") pod \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\" (UID: \"86f1cc30-a121-47c7-a7c6-f7b40a2d5247\") " Sep 08 16:38:10.521222 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.521197 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7673a92-62d0-451d-9c2d-4d1717d83ef1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7673a92-62d0-451d-9c2d-4d1717d83ef1" (UID: "a7673a92-62d0-451d-9c2d-4d1717d83ef1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:38:10.521222 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.521205 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86f1cc30-a121-47c7-a7c6-f7b40a2d5247" (UID: "86f1cc30-a121-47c7-a7c6-f7b40a2d5247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:38:10.521334 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.521321 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:10.521388 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.521340 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7673a92-62d0-451d-9c2d-4d1717d83ef1-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:10.523415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.523386 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-kube-api-access-5vf82" (OuterVolumeSpecName: "kube-api-access-5vf82") pod "86f1cc30-a121-47c7-a7c6-f7b40a2d5247" (UID: "86f1cc30-a121-47c7-a7c6-f7b40a2d5247"). InnerVolumeSpecName "kube-api-access-5vf82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:38:10.523530 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.523479 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7673a92-62d0-451d-9c2d-4d1717d83ef1-kube-api-access-d5rx8" (OuterVolumeSpecName: "kube-api-access-d5rx8") pod "a7673a92-62d0-451d-9c2d-4d1717d83ef1" (UID: "a7673a92-62d0-451d-9c2d-4d1717d83ef1"). InnerVolumeSpecName "kube-api-access-d5rx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:38:10.622611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.622577 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vf82\" (UniqueName: \"kubernetes.io/projected/86f1cc30-a121-47c7-a7c6-f7b40a2d5247-kube-api-access-5vf82\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:10.622611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.622606 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5rx8\" (UniqueName: \"kubernetes.io/projected/a7673a92-62d0-451d-9c2d-4d1717d83ef1-kube-api-access-d5rx8\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:10.769971 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.769881 2567 generic.go:356] "Generic (PLEG): container finished" podID="86f1cc30-a121-47c7-a7c6-f7b40a2d5247" containerID="f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f" exitCode=0 Sep 08 16:38:10.769971 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.769926 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" Sep 08 16:38:10.770247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.769966 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" event={"ID":"86f1cc30-a121-47c7-a7c6-f7b40a2d5247","Type":"ContainerDied","Data":"f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f"} Sep 08 16:38:10.770247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.770015 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" event={"ID":"86f1cc30-a121-47c7-a7c6-f7b40a2d5247","Type":"ContainerDied","Data":"c35f5bf615b8fe3e48490aed3eb080d6251e454cd43d74f0dfd1e1215b213a02"} Sep 08 16:38:10.770247 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.770070 2567 scope.go:117] "RemoveContainer" containerID="f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f" Sep 08 16:38:10.771430 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.771407 2567 generic.go:356] "Generic (PLEG): container finished" podID="a7673a92-62d0-451d-9c2d-4d1717d83ef1" containerID="a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6" exitCode=0 Sep 08 16:38:10.771562 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.771432 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" event={"ID":"a7673a92-62d0-451d-9c2d-4d1717d83ef1","Type":"ContainerDied","Data":"a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6"} Sep 08 16:38:10.771562 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.771459 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" Sep 08 16:38:10.771562 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.771478 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v" event={"ID":"a7673a92-62d0-451d-9c2d-4d1717d83ef1","Type":"ContainerDied","Data":"91a2898350885b5e54228c482503b379fc08078b760efc8979e878431da03d1a"} Sep 08 16:38:10.779692 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.779674 2567 scope.go:117] "RemoveContainer" containerID="f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f" Sep 08 16:38:10.779951 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:38:10.779932 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f\": container with ID starting with f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f not found: ID does not exist" containerID="f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f" Sep 08 16:38:10.780054 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.779955 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f"} err="failed to get container status \"f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f\": rpc error: code = NotFound desc = could not find container \"f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f\": container with ID starting with f16b204e8624ba70765f82a5399384b855ea67b71cfe47cda941b0f32617a04f not found: ID does not exist" Sep 08 16:38:10.780054 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.779969 2567 scope.go:117] "RemoveContainer" containerID="a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6" Sep 08 16:38:10.788391 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.788373 2567 scope.go:117] "RemoveContainer" containerID="a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6" Sep 08 16:38:10.788635 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:38:10.788617 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6\": container with ID starting with a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6 not found: ID does not exist" containerID="a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6" Sep 08 16:38:10.788679 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.788640 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6"} err="failed to get container status \"a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6\": rpc error: code = NotFound desc = could not find container \"a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6\": container with ID starting with a3e694ba981494bb4487ba60f1d885af38fbe983d1ce87cd0647f59913a042f6 not found: ID does not exist" Sep 08 16:38:10.794594 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.794572 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k"] Sep 08 16:38:10.797229 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.797209 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k"] Sep 08 16:38:10.808587 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.808568 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v"] Sep 08 16:38:10.811898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:10.811875 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-384f4f-5fc75bd998-6449v"] Sep 08 16:38:11.053607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:11.053531 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f1cc30-a121-47c7-a7c6-f7b40a2d5247" path="/var/lib/kubelet/pods/86f1cc30-a121-47c7-a7c6-f7b40a2d5247/volumes" Sep 08 16:38:11.053892 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:11.053880 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7673a92-62d0-451d-9c2d-4d1717d83ef1" path="/var/lib/kubelet/pods/a7673a92-62d0-451d-9c2d-4d1717d83ef1/volumes" Sep 08 16:38:11.054184 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:11.054172 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aef6ed3b-e41e-455c-90f2-d9dae60fbd9e" path="/var/lib/kubelet/pods/aef6ed3b-e41e-455c-90f2-d9dae60fbd9e/volumes" Sep 08 16:38:11.364048 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:11.363918 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-explicit-384f4f-7b4f8d854f-7h28k" podUID="86f1cc30-a121-47c7-a7c6-f7b40a2d5247" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.51:9004: i/o timeout" Sep 08 16:38:17.832318 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832288 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq"] Sep 08 16:38:17.832710 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832698 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f1cc30-a121-47c7-a7c6-f7b40a2d5247" containerName="payload-processing" Sep 08 16:38:17.832772 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832712 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f1cc30-a121-47c7-a7c6-f7b40a2d5247" containerName="payload-processing" Sep 08 16:38:17.832772 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832723 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef6ed3b-e41e-455c-90f2-d9dae60fbd9e" containerName="revoke-keys" Sep 08 16:38:17.832772 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832728 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef6ed3b-e41e-455c-90f2-d9dae60fbd9e" containerName="revoke-keys" Sep 08 16:38:17.832772 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832737 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7673a92-62d0-451d-9c2d-4d1717d83ef1" containerName="payload-pre-processing" Sep 08 16:38:17.832772 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832742 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7673a92-62d0-451d-9c2d-4d1717d83ef1" containerName="payload-pre-processing" Sep 08 16:38:17.832926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832799 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7673a92-62d0-451d-9c2d-4d1717d83ef1" containerName="payload-pre-processing" Sep 08 16:38:17.832926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832807 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef6ed3b-e41e-455c-90f2-d9dae60fbd9e" containerName="revoke-keys" Sep 08 16:38:17.832926 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.832815 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f1cc30-a121-47c7-a7c6-f7b40a2d5247" containerName="payload-processing" Sep 08 16:38:17.840738 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.840711 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.843573 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.843551 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 16:38:17.843823 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.843806 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-56547c-openshift-default-dockercfg-9v45d\"" Sep 08 16:38:17.854546 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.854522 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq"] Sep 08 16:38:17.891907 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.891875 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-certs\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.892064 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.891910 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-socket\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.892064 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.891945 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7lv\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-kube-api-access-dk7lv\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.892064 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.892013 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8af25a-3323-4e5a-8910-d9500bd91095-istiod-ca-cert\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.892064 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.892057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-token\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.892214 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.892083 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-podinfo\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.892214 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.892173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-envoy\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.892279 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.892230 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-data\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.892279 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.892272 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-credential-socket\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.993597 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.993563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-envoy\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.993782 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.993618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-data\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.993782 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.993660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-credential-socket\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.993782 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.993684 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-certs\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.993782 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.993701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-socket\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.993782 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.993736 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7lv\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-kube-api-access-dk7lv\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.994041 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.993813 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8af25a-3323-4e5a-8910-d9500bd91095-istiod-ca-cert\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.994041 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.993851 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-token\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.994041 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.993889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-podinfo\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.994203 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.994141 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-certs\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.994203 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.994172 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-credential-socket\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.994290 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.994236 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-data\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.994361 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.994338 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-socket\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.994627 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.994610 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8af25a-3323-4e5a-8910-d9500bd91095-istiod-ca-cert\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.996034 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.995999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-envoy\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:17.996408 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:17.996391 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-podinfo\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:18.003149 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:18.003125 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-token\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:18.003541 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:18.003519 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7lv\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-kube-api-access-dk7lv\") pod \"e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:18.150921 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:18.150825 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:18.433105 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:18.433078 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq"] Sep 08 16:38:18.434563 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:38:18.434532 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8af25a_3323_4e5a_8910_d9500bd91095.slice/crio-f5214de6567796047d722d194e0c4cce840e2978841c3b902d1c663e8a3d6123 WatchSource:0}: Error finding container f5214de6567796047d722d194e0c4cce840e2978841c3b902d1c663e8a3d6123: Status 404 returned error can't find the container with id f5214de6567796047d722d194e0c4cce840e2978841c3b902d1c663e8a3d6123 Sep 08 16:38:18.814912 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:18.814813 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" event={"ID":"3b8af25a-3323-4e5a-8910-d9500bd91095","Type":"ContainerStarted","Data":"f5214de6567796047d722d194e0c4cce840e2978841c3b902d1c663e8a3d6123"} Sep 08 16:38:21.455124 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:21.454380 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 16:38:21.455124 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:21.454491 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 16:38:21.455124 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:21.454528 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 16:38:21.833719 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:21.833620 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" event={"ID":"3b8af25a-3323-4e5a-8910-d9500bd91095","Type":"ContainerStarted","Data":"f66b1319be112e4759ed5f51f6e77fe1e15b97e194cfc8b039ccddedf720f414"} Sep 08 16:38:21.859399 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:21.859349 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" podStartSLOduration=1.842307056 podStartE2EDuration="4.859334281s" totalImagesPullingTime="3.017027225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:21.857462693 +0000 UTC m=+929.328449151" watchObservedRunningTime="2026-09-08 16:38:21.859334281 +0000 UTC m=+929.330320737" Sep 08 16:38:22.152098 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:22.151945 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:22.153513 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:22.153476 2567 patch_prober.go:28] interesting pod/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 08 16:38:22.153651 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:22.153544 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" podUID="3b8af25a-3323-4e5a-8910-d9500bd91095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 08 16:38:23.155772 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:23.155745 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:23.844765 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:23.844729 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:23.845717 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:23.845697 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:27.054843 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.054813 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw"] Sep 08 16:38:27.058716 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.058692 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.061288 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.061264 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-56547c\"/\"kube-root-ca.crt\"" Sep 08 16:38:27.062163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.062144 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-56547c\"/\"openshift-service-ca.crt\"" Sep 08 16:38:27.062261 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.062244 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-56547c\"/\"noten-model-56547c-kserve-self-signed-certs\"" Sep 08 16:38:27.062318 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.062272 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-56547c\"/\"default-dockercfg-w4q8r\"" Sep 08 16:38:27.067052 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.067003 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw"] Sep 08 16:38:27.182919 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.182889 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-model-cache\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.183102 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.182942 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfw4p\" (UniqueName: \"kubernetes.io/projected/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-kube-api-access-nfw4p\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.183102 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.183014 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-home\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.183102 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.183073 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tmp-dir\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.183218 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.183107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-dshm\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.183218 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.183144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tls-certs\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.284137 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.284091 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfw4p\" (UniqueName: \"kubernetes.io/projected/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-kube-api-access-nfw4p\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.284342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.284153 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-home\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.284342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.284176 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tmp-dir\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.284342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.284198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-dshm\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.284342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.284229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tls-certs\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.284342 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.284319 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-model-cache\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.284595 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.284571 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tmp-dir\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.284685 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.284664 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-home\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.284730 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.284692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-model-cache\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.286541 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.286514 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-dshm\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.286878 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.286863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tls-certs\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.292497 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.292471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfw4p\" (UniqueName: \"kubernetes.io/projected/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-kube-api-access-nfw4p\") pod \"noten-model-56547c-kserve-f44449759-nvkbw\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.369797 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.369718 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:27.642951 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.642924 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw"] Sep 08 16:38:27.644853 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:38:27.644828 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d62673f_d00e_43cd_bbf7_616b1fcb42d7.slice/crio-88386c627ba683fc97d7c2cb1aa07a62a0627b263e4152d43604830ceac3cf64 WatchSource:0}: Error finding container 88386c627ba683fc97d7c2cb1aa07a62a0627b263e4152d43604830ceac3cf64: Status 404 returned error can't find the container with id 88386c627ba683fc97d7c2cb1aa07a62a0627b263e4152d43604830ceac3cf64 Sep 08 16:38:27.867436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.867398 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" event={"ID":"2d62673f-d00e-43cd-bbf7-616b1fcb42d7","Type":"ContainerStarted","Data":"462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8"} Sep 08 16:38:27.867436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.867437 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" event={"ID":"2d62673f-d00e-43cd-bbf7-616b1fcb42d7","Type":"ContainerStarted","Data":"88386c627ba683fc97d7c2cb1aa07a62a0627b263e4152d43604830ceac3cf64"} Sep 08 16:38:27.887511 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:27.887456 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" podStartSLOduration=0.887441292 podStartE2EDuration="887.441292ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:27.885364024 +0000 UTC m=+935.356350504" watchObservedRunningTime="2026-09-08 16:38:27.887441292 +0000 UTC m=+935.358427748" Sep 08 16:38:30.566659 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:30.566625 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw"] Sep 08 16:38:30.705156 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:30.705123 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq"] Sep 08 16:38:30.705357 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:30.705333 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" podUID="3b8af25a-3323-4e5a-8910-d9500bd91095" containerName="istio-proxy" containerID="cri-o://f66b1319be112e4759ed5f51f6e77fe1e15b97e194cfc8b039ccddedf720f414" gracePeriod=30 Sep 08 16:38:35.908811 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:35.908775 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b8af25a-3323-4e5a-8910-d9500bd91095" containerID="f66b1319be112e4759ed5f51f6e77fe1e15b97e194cfc8b039ccddedf720f414" exitCode=0 Sep 08 16:38:35.909168 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:35.908844 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" event={"ID":"3b8af25a-3323-4e5a-8910-d9500bd91095","Type":"ContainerDied","Data":"f66b1319be112e4759ed5f51f6e77fe1e15b97e194cfc8b039ccddedf720f414"} Sep 08 16:38:35.953940 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:35.953916 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:36.071569 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071489 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-credential-socket\") pod \"3b8af25a-3323-4e5a-8910-d9500bd91095\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " Sep 08 16:38:36.071569 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071559 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-socket\") pod \"3b8af25a-3323-4e5a-8910-d9500bd91095\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " Sep 08 16:38:36.071763 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071586 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-podinfo\") pod \"3b8af25a-3323-4e5a-8910-d9500bd91095\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " Sep 08 16:38:36.071763 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071605 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-envoy\") pod \"3b8af25a-3323-4e5a-8910-d9500bd91095\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " Sep 08 16:38:36.071763 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071626 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-token\") pod \"3b8af25a-3323-4e5a-8910-d9500bd91095\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " Sep 08 16:38:36.071763 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071664 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-certs\") pod \"3b8af25a-3323-4e5a-8910-d9500bd91095\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " Sep 08 16:38:36.071763 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071708 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8af25a-3323-4e5a-8910-d9500bd91095-istiod-ca-cert\") pod \"3b8af25a-3323-4e5a-8910-d9500bd91095\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " Sep 08 16:38:36.072034 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071766 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-data\") pod \"3b8af25a-3323-4e5a-8910-d9500bd91095\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " Sep 08 16:38:36.072034 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071794 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk7lv\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-kube-api-access-dk7lv\") pod \"3b8af25a-3323-4e5a-8910-d9500bd91095\" (UID: \"3b8af25a-3323-4e5a-8910-d9500bd91095\") " Sep 08 16:38:36.072034 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071818 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b8af25a-3323-4e5a-8910-d9500bd91095" (UID: "3b8af25a-3323-4e5a-8910-d9500bd91095"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:38:36.072034 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.071955 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b8af25a-3323-4e5a-8910-d9500bd91095" (UID: "3b8af25a-3323-4e5a-8910-d9500bd91095"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:38:36.072270 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.072193 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8af25a-3323-4e5a-8910-d9500bd91095-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b8af25a-3323-4e5a-8910-d9500bd91095" (UID: "3b8af25a-3323-4e5a-8910-d9500bd91095"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:38:36.072270 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.072223 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b8af25a-3323-4e5a-8910-d9500bd91095" (UID: "3b8af25a-3323-4e5a-8910-d9500bd91095"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:38:36.072385 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.072283 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b8af25a-3323-4e5a-8910-d9500bd91095" (UID: "3b8af25a-3323-4e5a-8910-d9500bd91095"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:38:36.072385 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.072287 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8af25a-3323-4e5a-8910-d9500bd91095-istiod-ca-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:36.072385 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.072330 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-credential-socket\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:36.072385 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.072348 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-socket\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:36.073913 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.073892 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b8af25a-3323-4e5a-8910-d9500bd91095" (UID: "3b8af25a-3323-4e5a-8910-d9500bd91095"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:38:36.074208 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.074188 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-kube-api-access-dk7lv" (OuterVolumeSpecName: "kube-api-access-dk7lv") pod "3b8af25a-3323-4e5a-8910-d9500bd91095" (UID: "3b8af25a-3323-4e5a-8910-d9500bd91095"). InnerVolumeSpecName "kube-api-access-dk7lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:38:36.074459 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.074441 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b8af25a-3323-4e5a-8910-d9500bd91095" (UID: "3b8af25a-3323-4e5a-8910-d9500bd91095"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:38:36.074500 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.074449 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b8af25a-3323-4e5a-8910-d9500bd91095" (UID: "3b8af25a-3323-4e5a-8910-d9500bd91095"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:38:36.173202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.173167 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-podinfo\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:36.173202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.173196 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-envoy\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:36.173202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.173207 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-token\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:36.173441 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.173214 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-workload-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:36.173441 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.173222 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8af25a-3323-4e5a-8910-d9500bd91095-istio-data\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:36.173441 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.173230 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk7lv\" (UniqueName: \"kubernetes.io/projected/3b8af25a-3323-4e5a-8910-d9500bd91095-kube-api-access-dk7lv\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:36.915960 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.915928 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" Sep 08 16:38:36.916378 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.915959 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq" event={"ID":"3b8af25a-3323-4e5a-8910-d9500bd91095","Type":"ContainerDied","Data":"f5214de6567796047d722d194e0c4cce840e2978841c3b902d1c663e8a3d6123"} Sep 08 16:38:36.916378 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.916033 2567 scope.go:117] "RemoveContainer" containerID="f66b1319be112e4759ed5f51f6e77fe1e15b97e194cfc8b039ccddedf720f414" Sep 08 16:38:36.941947 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.941922 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq"] Sep 08 16:38:36.945900 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:36.945874 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-56547c-openshift-default-5496d9b88d-hw4bq"] Sep 08 16:38:37.054079 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:37.054038 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b8af25a-3323-4e5a-8910-d9500bd91095" path="/var/lib/kubelet/pods/3b8af25a-3323-4e5a-8910-d9500bd91095/volumes" Sep 08 16:38:37.370766 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:37.370656 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:45.587658 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.587615 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" podUID="2d62673f-d00e-43cd-bbf7-616b1fcb42d7" containerName="main" containerID="cri-o://462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8" gracePeriod=45 Sep 08 16:38:45.843664 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.843609 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:45.865405 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.865379 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-model-cache\") pod \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " Sep 08 16:38:45.865548 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.865450 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfw4p\" (UniqueName: \"kubernetes.io/projected/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-kube-api-access-nfw4p\") pod \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " Sep 08 16:38:45.865548 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.865490 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-dshm\") pod \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " Sep 08 16:38:45.865688 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.865565 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-home\") pod \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " Sep 08 16:38:45.865688 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.865624 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tls-certs\") pod \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " Sep 08 16:38:45.865793 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.865696 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tmp-dir\") pod \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\" (UID: \"2d62673f-d00e-43cd-bbf7-616b1fcb42d7\") " Sep 08 16:38:45.865793 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.865691 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d62673f-d00e-43cd-bbf7-616b1fcb42d7" (UID: "2d62673f-d00e-43cd-bbf7-616b1fcb42d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:38:45.865901 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.865871 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-home" (OuterVolumeSpecName: "home") pod "2d62673f-d00e-43cd-bbf7-616b1fcb42d7" (UID: "2d62673f-d00e-43cd-bbf7-616b1fcb42d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:38:45.866174 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.866102 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d62673f-d00e-43cd-bbf7-616b1fcb42d7" (UID: "2d62673f-d00e-43cd-bbf7-616b1fcb42d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:38:45.866174 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.866108 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:45.866174 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.866152 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:45.868171 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.868137 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-dshm" (OuterVolumeSpecName: "dshm") pod "2d62673f-d00e-43cd-bbf7-616b1fcb42d7" (UID: "2d62673f-d00e-43cd-bbf7-616b1fcb42d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:38:45.868304 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.868140 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d62673f-d00e-43cd-bbf7-616b1fcb42d7" (UID: "2d62673f-d00e-43cd-bbf7-616b1fcb42d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:38:45.868304 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.868239 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-kube-api-access-nfw4p" (OuterVolumeSpecName: "kube-api-access-nfw4p") pod "2d62673f-d00e-43cd-bbf7-616b1fcb42d7" (UID: "2d62673f-d00e-43cd-bbf7-616b1fcb42d7"). InnerVolumeSpecName "kube-api-access-nfw4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:38:45.965494 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.965466 2567 generic.go:356] "Generic (PLEG): container finished" podID="2d62673f-d00e-43cd-bbf7-616b1fcb42d7" containerID="462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8" exitCode=0 Sep 08 16:38:45.965630 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.965514 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" event={"ID":"2d62673f-d00e-43cd-bbf7-616b1fcb42d7","Type":"ContainerDied","Data":"462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8"} Sep 08 16:38:45.965630 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.965515 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" Sep 08 16:38:45.965630 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.965552 2567 scope.go:117] "RemoveContainer" containerID="462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8" Sep 08 16:38:45.965630 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.965541 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw" event={"ID":"2d62673f-d00e-43cd-bbf7-616b1fcb42d7","Type":"ContainerDied","Data":"88386c627ba683fc97d7c2cb1aa07a62a0627b263e4152d43604830ceac3cf64"} Sep 08 16:38:45.966745 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.966663 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:45.966745 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.966682 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:45.966745 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.966697 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:45.966745 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.966709 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfw4p\" (UniqueName: \"kubernetes.io/projected/2d62673f-d00e-43cd-bbf7-616b1fcb42d7-kube-api-access-nfw4p\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:45.974751 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.974734 2567 scope.go:117] "RemoveContainer" containerID="462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8" Sep 08 16:38:45.975056 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:38:45.975013 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8\": container with ID starting with 462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8 not found: ID does not exist" containerID="462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8" Sep 08 16:38:45.975123 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.975064 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8"} err="failed to get container status \"462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8\": rpc error: code = NotFound desc = could not find container \"462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8\": container with ID starting with 462a928d9f87ecec11dc5fbdb203acf323f11bc8541b1d3ca2027056acdfeeb8 not found: ID does not exist" Sep 08 16:38:45.986593 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.986569 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw"] Sep 08 16:38:45.990368 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:45.990347 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-56547c/noten-model-56547c-kserve-f44449759-nvkbw"] Sep 08 16:38:47.052820 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:47.052788 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d62673f-d00e-43cd-bbf7-616b1fcb42d7" path="/var/lib/kubelet/pods/2d62673f-d00e-43cd-bbf7-616b1fcb42d7/volumes" Sep 08 16:38:51.995809 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:51.995777 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27"] Sep 08 16:38:51.996239 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:51.996225 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8af25a-3323-4e5a-8910-d9500bd91095" containerName="istio-proxy" Sep 08 16:38:51.996291 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:51.996241 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8af25a-3323-4e5a-8910-d9500bd91095" containerName="istio-proxy" Sep 08 16:38:51.996291 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:51.996250 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d62673f-d00e-43cd-bbf7-616b1fcb42d7" containerName="main" Sep 08 16:38:51.996291 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:51.996255 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d62673f-d00e-43cd-bbf7-616b1fcb42d7" containerName="main" Sep 08 16:38:51.996403 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:51.996317 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b8af25a-3323-4e5a-8910-d9500bd91095" containerName="istio-proxy" Sep 08 16:38:51.996403 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:51.996330 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d62673f-d00e-43cd-bbf7-616b1fcb42d7" containerName="main" Sep 08 16:38:51.999616 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:51.999599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" Sep 08 16:38:52.002146 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:52.002124 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:38:52.002274 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:52.002125 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzslb\"" Sep 08 16:38:52.008120 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:52.008089 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27"] Sep 08 16:38:52.023532 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:52.023500 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0b20ec-2805-4d94-9905-af21f928e72b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27\" (UID: \"2c0b20ec-2805-4d94-9905-af21f928e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" Sep 08 16:38:52.124969 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:52.124928 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0b20ec-2805-4d94-9905-af21f928e72b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27\" (UID: \"2c0b20ec-2805-4d94-9905-af21f928e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" Sep 08 16:38:52.125554 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:52.125534 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0b20ec-2805-4d94-9905-af21f928e72b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27\" (UID: \"2c0b20ec-2805-4d94-9905-af21f928e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" Sep 08 16:38:52.311797 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:52.311702 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" Sep 08 16:38:52.583730 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:52.583703 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27"] Sep 08 16:38:52.586726 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:38:52.586692 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0b20ec_2805_4d94_9905_af21f928e72b.slice/crio-18ba9c96ce4b9f58b14e3e79399b7efba705ff50609189c456f8a14683856b85 WatchSource:0}: Error finding container 18ba9c96ce4b9f58b14e3e79399b7efba705ff50609189c456f8a14683856b85: Status 404 returned error can't find the container with id 18ba9c96ce4b9f58b14e3e79399b7efba705ff50609189c456f8a14683856b85 Sep 08 16:38:53.003872 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:53.003832 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" event={"ID":"2c0b20ec-2805-4d94-9905-af21f928e72b","Type":"ContainerStarted","Data":"73a543356745ccf979b09be21f74d9ac771913b553c40b8b394d7813e35376d3"} Sep 08 16:38:53.004279 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:53.003918 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" event={"ID":"2c0b20ec-2805-4d94-9905-af21f928e72b","Type":"ContainerStarted","Data":"18ba9c96ce4b9f58b14e3e79399b7efba705ff50609189c456f8a14683856b85"} Sep 08 16:38:53.024589 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:53.024537 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" podStartSLOduration=2.02452214 podStartE2EDuration="2.02452214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:53.022301894 +0000 UTC m=+960.493288353" watchObservedRunningTime="2026-09-08 16:38:53.02452214 +0000 UTC m=+960.495508596" Sep 08 16:38:54.010575 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:54.010540 2567 generic.go:356] "Generic (PLEG): container finished" podID="2c0b20ec-2805-4d94-9905-af21f928e72b" containerID="73a543356745ccf979b09be21f74d9ac771913b553c40b8b394d7813e35376d3" exitCode=0 Sep 08 16:38:54.010976 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:54.010625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" event={"ID":"2c0b20ec-2805-4d94-9905-af21f928e72b","Type":"ContainerDied","Data":"73a543356745ccf979b09be21f74d9ac771913b553c40b8b394d7813e35376d3"} Sep 08 16:38:55.178545 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:55.178519 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" Sep 08 16:38:55.254153 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:55.254121 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0b20ec-2805-4d94-9905-af21f928e72b-maas-api-service-ca\") pod \"2c0b20ec-2805-4d94-9905-af21f928e72b\" (UID: \"2c0b20ec-2805-4d94-9905-af21f928e72b\") " Sep 08 16:38:55.254454 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:55.254432 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0b20ec-2805-4d94-9905-af21f928e72b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c0b20ec-2805-4d94-9905-af21f928e72b" (UID: "2c0b20ec-2805-4d94-9905-af21f928e72b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:38:55.355883 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:55.355789 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0b20ec-2805-4d94-9905-af21f928e72b-maas-api-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:38:56.023346 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:56.023312 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" Sep 08 16:38:56.023526 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:56.023346 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27" event={"ID":"2c0b20ec-2805-4d94-9905-af21f928e72b","Type":"ContainerDied","Data":"18ba9c96ce4b9f58b14e3e79399b7efba705ff50609189c456f8a14683856b85"} Sep 08 16:38:56.023526 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:38:56.023400 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18ba9c96ce4b9f58b14e3e79399b7efba705ff50609189c456f8a14683856b85" Sep 08 16:39:02.506897 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.506859 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf"] Sep 08 16:39:02.507832 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.507429 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" podUID="19577729-a864-4ada-be14-f753ba3e785f" containerName="payload-pre-processing" containerID="cri-o://230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775" gracePeriod=30 Sep 08 16:39:02.697088 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.697008 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27"] Sep 08 16:39:02.699819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.699779 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff43b8-dbr27"] Sep 08 16:39:02.768134 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.768065 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:39:02.832429 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.832394 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm4fs\" (UniqueName: \"kubernetes.io/projected/19577729-a864-4ada-be14-f753ba3e785f-kube-api-access-hm4fs\") pod \"19577729-a864-4ada-be14-f753ba3e785f\" (UID: \"19577729-a864-4ada-be14-f753ba3e785f\") " Sep 08 16:39:02.832624 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.832440 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19577729-a864-4ada-be14-f753ba3e785f-plugins-config-volume\") pod \"19577729-a864-4ada-be14-f753ba3e785f\" (UID: \"19577729-a864-4ada-be14-f753ba3e785f\") " Sep 08 16:39:02.832833 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.832807 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19577729-a864-4ada-be14-f753ba3e785f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19577729-a864-4ada-be14-f753ba3e785f" (UID: "19577729-a864-4ada-be14-f753ba3e785f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:39:02.834733 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.834699 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19577729-a864-4ada-be14-f753ba3e785f-kube-api-access-hm4fs" (OuterVolumeSpecName: "kube-api-access-hm4fs") pod "19577729-a864-4ada-be14-f753ba3e785f" (UID: "19577729-a864-4ada-be14-f753ba3e785f"). InnerVolumeSpecName "kube-api-access-hm4fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:39:02.933273 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.933232 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4fs\" (UniqueName: \"kubernetes.io/projected/19577729-a864-4ada-be14-f753ba3e785f-kube-api-access-hm4fs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:39:02.933273 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:02.933264 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19577729-a864-4ada-be14-f753ba3e785f-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:39:03.053197 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.053111 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c0b20ec-2805-4d94-9905-af21f928e72b" path="/var/lib/kubelet/pods/2c0b20ec-2805-4d94-9905-af21f928e72b/volumes" Sep 08 16:39:03.063069 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.063008 2567 generic.go:356] "Generic (PLEG): container finished" podID="19577729-a864-4ada-be14-f753ba3e785f" containerID="230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775" exitCode=0 Sep 08 16:39:03.063198 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.063086 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" Sep 08 16:39:03.063198 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.063089 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" event={"ID":"19577729-a864-4ada-be14-f753ba3e785f","Type":"ContainerDied","Data":"230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775"} Sep 08 16:39:03.063198 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.063154 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf" event={"ID":"19577729-a864-4ada-be14-f753ba3e785f","Type":"ContainerDied","Data":"70066d8af32d6921ab7eb8e301e5a398b736cd65036018ae595da4176333880c"} Sep 08 16:39:03.063198 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.063177 2567 scope.go:117] "RemoveContainer" containerID="230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775" Sep 08 16:39:03.072346 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.072324 2567 scope.go:117] "RemoveContainer" containerID="230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775" Sep 08 16:39:03.072647 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:39:03.072627 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775\": container with ID starting with 230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775 not found: ID does not exist" containerID="230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775" Sep 08 16:39:03.072690 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.072655 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775"} err="failed to get container status \"230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775\": rpc error: code = NotFound desc = could not find container \"230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775\": container with ID starting with 230f650b2fec996d3df2037ca42ab470d28199e52e21bdbe88a0bdbfda0eb775 not found: ID does not exist" Sep 08 16:39:03.079794 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.079769 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf"] Sep 08 16:39:03.083203 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:03.083179 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff43b8-666b566bdb-47dxf"] Sep 08 16:39:05.053289 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:05.053256 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19577729-a864-4ada-be14-f753ba3e785f" path="/var/lib/kubelet/pods/19577729-a864-4ada-be14-f753ba3e785f/volumes" Sep 08 16:39:09.673546 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.673510 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6"] Sep 08 16:39:09.674090 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.674068 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19577729-a864-4ada-be14-f753ba3e785f" containerName="payload-pre-processing" Sep 08 16:39:09.674090 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.674088 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="19577729-a864-4ada-be14-f753ba3e785f" containerName="payload-pre-processing" Sep 08 16:39:09.674202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.674116 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0b20ec-2805-4d94-9905-af21f928e72b" containerName="revoke-keys" Sep 08 16:39:09.674202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.674122 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0b20ec-2805-4d94-9905-af21f928e72b" containerName="revoke-keys" Sep 08 16:39:09.674202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.674183 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0b20ec-2805-4d94-9905-af21f928e72b" containerName="revoke-keys" Sep 08 16:39:09.674202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.674194 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="19577729-a864-4ada-be14-f753ba3e785f" containerName="payload-pre-processing" Sep 08 16:39:09.677522 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.677508 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" Sep 08 16:39:09.680533 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.680503 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzslb\"" Sep 08 16:39:09.680652 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.680532 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:39:09.687065 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.687036 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6"] Sep 08 16:39:09.802008 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.801968 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df1234ae-050c-46f8-a806-762bb3e68eec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6\" (UID: \"df1234ae-050c-46f8-a806-762bb3e68eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" Sep 08 16:39:09.902995 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.902961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df1234ae-050c-46f8-a806-762bb3e68eec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6\" (UID: \"df1234ae-050c-46f8-a806-762bb3e68eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" Sep 08 16:39:09.903590 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.903567 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df1234ae-050c-46f8-a806-762bb3e68eec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6\" (UID: \"df1234ae-050c-46f8-a806-762bb3e68eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" Sep 08 16:39:09.989500 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:09.989403 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" Sep 08 16:39:10.262942 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:10.262918 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6"] Sep 08 16:39:10.265204 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:39:10.265170 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1234ae_050c_46f8_a806_762bb3e68eec.slice/crio-b663853161239eded05b61e776ef2d8ff054eb522623bf5faf077aa7d5761c11 WatchSource:0}: Error finding container b663853161239eded05b61e776ef2d8ff054eb522623bf5faf077aa7d5761c11: Status 404 returned error can't find the container with id b663853161239eded05b61e776ef2d8ff054eb522623bf5faf077aa7d5761c11 Sep 08 16:39:11.106238 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:11.106211 2567 generic.go:356] "Generic (PLEG): container finished" podID="df1234ae-050c-46f8-a806-762bb3e68eec" containerID="19f99a67815095f60fb0f2ffa53b961105e660b4d2adb586176cdedf750197f0" exitCode=0 Sep 08 16:39:11.106594 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:11.106244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" event={"ID":"df1234ae-050c-46f8-a806-762bb3e68eec","Type":"ContainerDied","Data":"19f99a67815095f60fb0f2ffa53b961105e660b4d2adb586176cdedf750197f0"} Sep 08 16:39:11.106594 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:11.106280 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" event={"ID":"df1234ae-050c-46f8-a806-762bb3e68eec","Type":"ContainerStarted","Data":"b663853161239eded05b61e776ef2d8ff054eb522623bf5faf077aa7d5761c11"} Sep 08 16:39:12.243416 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:12.243390 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" Sep 08 16:39:12.427110 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:12.427074 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df1234ae-050c-46f8-a806-762bb3e68eec-maas-api-service-ca\") pod \"df1234ae-050c-46f8-a806-762bb3e68eec\" (UID: \"df1234ae-050c-46f8-a806-762bb3e68eec\") " Sep 08 16:39:12.427474 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:12.427453 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1234ae-050c-46f8-a806-762bb3e68eec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df1234ae-050c-46f8-a806-762bb3e68eec" (UID: "df1234ae-050c-46f8-a806-762bb3e68eec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:39:12.527802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:12.527766 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df1234ae-050c-46f8-a806-762bb3e68eec-maas-api-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:39:13.117978 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:13.117946 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" Sep 08 16:39:13.118151 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:13.117984 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6" event={"ID":"df1234ae-050c-46f8-a806-762bb3e68eec","Type":"ContainerDied","Data":"b663853161239eded05b61e776ef2d8ff054eb522623bf5faf077aa7d5761c11"} Sep 08 16:39:13.118151 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:13.118052 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b663853161239eded05b61e776ef2d8ff054eb522623bf5faf077aa7d5761c11" Sep 08 16:39:20.451845 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.451794 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb"] Sep 08 16:39:20.452615 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.452559 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" podUID="bb54d10d-0859-4fad-8077-ed62494a4632" containerName="payload-processing" containerID="cri-o://c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d" gracePeriod=30 Sep 08 16:39:20.465942 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.465916 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m"] Sep 08 16:39:20.466132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.466113 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" podUID="73328598-385a-4df6-a835-212033ffd056" containerName="payload-pre-processing" containerID="cri-o://e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4" gracePeriod=30 Sep 08 16:39:20.652278 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.652174 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6"] Sep 08 16:39:20.657546 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.657517 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-870dc0-kccn6"] Sep 08 16:39:20.745838 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.745814 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:39:20.748581 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.748558 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:39:20.805123 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.805087 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2lqj\" (UniqueName: \"kubernetes.io/projected/73328598-385a-4df6-a835-212033ffd056-kube-api-access-k2lqj\") pod \"73328598-385a-4df6-a835-212033ffd056\" (UID: \"73328598-385a-4df6-a835-212033ffd056\") " Sep 08 16:39:20.805123 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.805129 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54d10d-0859-4fad-8077-ed62494a4632-plugins-config-volume\") pod \"bb54d10d-0859-4fad-8077-ed62494a4632\" (UID: \"bb54d10d-0859-4fad-8077-ed62494a4632\") " Sep 08 16:39:20.805361 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.805173 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73328598-385a-4df6-a835-212033ffd056-plugins-config-volume\") pod \"73328598-385a-4df6-a835-212033ffd056\" (UID: \"73328598-385a-4df6-a835-212033ffd056\") " Sep 08 16:39:20.805361 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.805303 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfn4t\" (UniqueName: \"kubernetes.io/projected/bb54d10d-0859-4fad-8077-ed62494a4632-kube-api-access-dfn4t\") pod \"bb54d10d-0859-4fad-8077-ed62494a4632\" (UID: \"bb54d10d-0859-4fad-8077-ed62494a4632\") " Sep 08 16:39:20.805564 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.805538 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb54d10d-0859-4fad-8077-ed62494a4632-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb54d10d-0859-4fad-8077-ed62494a4632" (UID: "bb54d10d-0859-4fad-8077-ed62494a4632"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:39:20.805648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.805537 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73328598-385a-4df6-a835-212033ffd056-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73328598-385a-4df6-a835-212033ffd056" (UID: "73328598-385a-4df6-a835-212033ffd056"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:39:20.805703 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.805654 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb54d10d-0859-4fad-8077-ed62494a4632-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:39:20.805703 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.805672 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73328598-385a-4df6-a835-212033ffd056-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:39:20.807521 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.807490 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb54d10d-0859-4fad-8077-ed62494a4632-kube-api-access-dfn4t" (OuterVolumeSpecName: "kube-api-access-dfn4t") pod "bb54d10d-0859-4fad-8077-ed62494a4632" (UID: "bb54d10d-0859-4fad-8077-ed62494a4632"). InnerVolumeSpecName "kube-api-access-dfn4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:39:20.807521 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.807513 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73328598-385a-4df6-a835-212033ffd056-kube-api-access-k2lqj" (OuterVolumeSpecName: "kube-api-access-k2lqj") pod "73328598-385a-4df6-a835-212033ffd056" (UID: "73328598-385a-4df6-a835-212033ffd056"). InnerVolumeSpecName "kube-api-access-k2lqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:39:20.906942 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.906910 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2lqj\" (UniqueName: \"kubernetes.io/projected/73328598-385a-4df6-a835-212033ffd056-kube-api-access-k2lqj\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:39:20.906942 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:20.906936 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfn4t\" (UniqueName: \"kubernetes.io/projected/bb54d10d-0859-4fad-8077-ed62494a4632-kube-api-access-dfn4t\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:39:21.053088 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.052986 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df1234ae-050c-46f8-a806-762bb3e68eec" path="/var/lib/kubelet/pods/df1234ae-050c-46f8-a806-762bb3e68eec/volumes" Sep 08 16:39:21.160213 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.160174 2567 generic.go:356] "Generic (PLEG): container finished" podID="73328598-385a-4df6-a835-212033ffd056" containerID="e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4" exitCode=0 Sep 08 16:39:21.160396 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.160224 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" Sep 08 16:39:21.160396 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.160269 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" event={"ID":"73328598-385a-4df6-a835-212033ffd056","Type":"ContainerDied","Data":"e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4"} Sep 08 16:39:21.160396 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.160320 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m" event={"ID":"73328598-385a-4df6-a835-212033ffd056","Type":"ContainerDied","Data":"32aa9515b25f988f77464e2b0d897f1753a9d22bd6e0cc1faf41a9ae8b221b8a"} Sep 08 16:39:21.160396 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.160348 2567 scope.go:117] "RemoveContainer" containerID="e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4" Sep 08 16:39:21.161916 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.161882 2567 generic.go:356] "Generic (PLEG): container finished" podID="bb54d10d-0859-4fad-8077-ed62494a4632" containerID="c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d" exitCode=0 Sep 08 16:39:21.162055 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.161918 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" event={"ID":"bb54d10d-0859-4fad-8077-ed62494a4632","Type":"ContainerDied","Data":"c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d"} Sep 08 16:39:21.162055 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.161926 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" Sep 08 16:39:21.162055 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.161957 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb" event={"ID":"bb54d10d-0859-4fad-8077-ed62494a4632","Type":"ContainerDied","Data":"6768aef25e8ba2e83a9ff622db3ca932637e39f9b92dcb3a44b2a8a13056c312"} Sep 08 16:39:21.169296 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.169276 2567 scope.go:117] "RemoveContainer" containerID="e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4" Sep 08 16:39:21.169571 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:39:21.169552 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4\": container with ID starting with e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4 not found: ID does not exist" containerID="e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4" Sep 08 16:39:21.169633 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.169579 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4"} err="failed to get container status \"e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4\": rpc error: code = NotFound desc = could not find container \"e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4\": container with ID starting with e0c183e8d67ef1843f402d31d450d1eca6e0c6cd37b9d70b8e4ade5a0a65d2e4 not found: ID does not exist" Sep 08 16:39:21.169633 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.169596 2567 scope.go:117] "RemoveContainer" containerID="c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d" Sep 08 16:39:21.177440 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.177426 2567 scope.go:117] "RemoveContainer" containerID="c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d" Sep 08 16:39:21.177692 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:39:21.177669 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d\": container with ID starting with c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d not found: ID does not exist" containerID="c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d" Sep 08 16:39:21.177745 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.177697 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d"} err="failed to get container status \"c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d\": rpc error: code = NotFound desc = could not find container \"c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d\": container with ID starting with c5cde61138cf6ceae144c52396ec55fa8e1a9190b9de58166940de716ae0da2d not found: ID does not exist" Sep 08 16:39:21.183040 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.183005 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m"] Sep 08 16:39:21.186802 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.186783 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-870dc0-79dc7b5894-fdg2m"] Sep 08 16:39:21.206087 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.206068 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb"] Sep 08 16:39:21.212056 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:21.212012 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-870dc0-8647bd7df9-fx8pb"] Sep 08 16:39:23.053116 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:23.053075 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73328598-385a-4df6-a835-212033ffd056" path="/var/lib/kubelet/pods/73328598-385a-4df6-a835-212033ffd056/volumes" Sep 08 16:39:23.053624 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:23.053604 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb54d10d-0859-4fad-8077-ed62494a4632" path="/var/lib/kubelet/pods/bb54d10d-0859-4fad-8077-ed62494a4632/volumes" Sep 08 16:39:25.344670 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.344638 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7"] Sep 08 16:39:25.345645 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.345611 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb54d10d-0859-4fad-8077-ed62494a4632" containerName="payload-processing" Sep 08 16:39:25.345645 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.345644 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb54d10d-0859-4fad-8077-ed62494a4632" containerName="payload-processing" Sep 08 16:39:25.345859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.345720 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df1234ae-050c-46f8-a806-762bb3e68eec" containerName="revoke-keys" Sep 08 16:39:25.345859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.345729 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="df1234ae-050c-46f8-a806-762bb3e68eec" containerName="revoke-keys" Sep 08 16:39:25.345859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.345739 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73328598-385a-4df6-a835-212033ffd056" containerName="payload-pre-processing" Sep 08 16:39:25.345859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.345748 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="73328598-385a-4df6-a835-212033ffd056" containerName="payload-pre-processing" Sep 08 16:39:25.346103 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.346085 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb54d10d-0859-4fad-8077-ed62494a4632" containerName="payload-processing" Sep 08 16:39:25.346270 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.346249 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="df1234ae-050c-46f8-a806-762bb3e68eec" containerName="revoke-keys" Sep 08 16:39:25.346355 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.346279 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="73328598-385a-4df6-a835-212033ffd056" containerName="payload-pre-processing" Sep 08 16:39:25.352238 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.352207 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:25.355803 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.355775 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a69d4afb-dockercfg-c5znk\"" Sep 08 16:39:25.355924 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.355775 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a69d4afb\"" Sep 08 16:39:25.357551 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.357528 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7"] Sep 08 16:39:25.449840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.449802 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpgtc\" (UniqueName: \"kubernetes.io/projected/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-kube-api-access-vpgtc\") pod \"payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7\" (UID: \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:25.450048 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.449891 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7\" (UID: \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:25.550990 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.550949 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpgtc\" (UniqueName: \"kubernetes.io/projected/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-kube-api-access-vpgtc\") pod \"payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7\" (UID: \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:25.551163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.551061 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7\" (UID: \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:25.551706 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.551687 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7\" (UID: \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:25.561233 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.561205 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpgtc\" (UniqueName: \"kubernetes.io/projected/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-kube-api-access-vpgtc\") pod \"payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7\" (UID: \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:25.664168 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.664138 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:25.940127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:25.940098 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7"] Sep 08 16:39:25.942540 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:39:25.942511 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61129e89_3ce1_42a6_bf31_1f4e841a4fb0.slice/crio-f09b7971752143f3e21aa4fcc7fac26ae72251666313a7853c07f7746f0654d6 WatchSource:0}: Error finding container f09b7971752143f3e21aa4fcc7fac26ae72251666313a7853c07f7746f0654d6: Status 404 returned error can't find the container with id f09b7971752143f3e21aa4fcc7fac26ae72251666313a7853c07f7746f0654d6 Sep 08 16:39:26.192495 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:26.192403 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" event={"ID":"61129e89-3ce1-42a6-bf31-1f4e841a4fb0","Type":"ContainerStarted","Data":"35102566b1fc929a3c9663933915d8dbad91e134ca0b7a71f622bf06bb90d4b3"} Sep 08 16:39:26.192495 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:26.192453 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" event={"ID":"61129e89-3ce1-42a6-bf31-1f4e841a4fb0","Type":"ContainerStarted","Data":"f09b7971752143f3e21aa4fcc7fac26ae72251666313a7853c07f7746f0654d6"} Sep 08 16:39:26.192699 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:26.192538 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:26.212001 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:26.211947 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" podStartSLOduration=1.211928872 podStartE2EDuration="1.211928872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:39:26.208686535 +0000 UTC m=+993.679672997" watchObservedRunningTime="2026-09-08 16:39:26.211928872 +0000 UTC m=+993.682915330" Sep 08 16:39:28.349323 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.349291 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt"] Sep 08 16:39:28.352726 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.352701 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.355221 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.355185 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 16:39:28.355365 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.355344 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a4942b-openshift-default-dockercfg-k2p8f\"" Sep 08 16:39:28.365939 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.365917 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt"] Sep 08 16:39:28.478117 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.478074 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdlr\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-kube-api-access-jhdlr\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.478289 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.478130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f919b7-2340-43d5-8a2f-5d21c06586d8-istiod-ca-cert\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.478289 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.478179 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-credential-socket\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.478289 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.478214 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-socket\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.478289 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.478269 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-envoy\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.478430 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.478305 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-podinfo\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.478430 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.478334 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-certs\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.478430 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.478369 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-token\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.478525 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.478422 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-data\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.579689 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.579650 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-certs\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.579689 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.579694 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-token\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.579903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.579720 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-data\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.579903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.579829 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdlr\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-kube-api-access-jhdlr\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.579903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.579876 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f919b7-2340-43d5-8a2f-5d21c06586d8-istiod-ca-cert\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.580047 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.579908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-credential-socket\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.580047 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.579945 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-socket\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.580160 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.580112 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-certs\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.580160 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.580141 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-data\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.580348 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.580324 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-socket\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.580419 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.580376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-envoy\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.580471 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.580419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-podinfo\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.580471 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.580374 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-credential-socket\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.580667 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.580648 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f919b7-2340-43d5-8a2f-5d21c06586d8-istiod-ca-cert\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.582377 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.582355 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-envoy\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.582778 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.582760 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-podinfo\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.596686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.596663 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-token\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.597108 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.597086 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdlr\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-kube-api-access-jhdlr\") pod \"e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.664392 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.664365 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:28.945204 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.945178 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt"] Sep 08 16:39:28.947372 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:39:28.947339 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f919b7_2340_43d5_8a2f_5d21c06586d8.slice/crio-5c785ed5c92bd8eae23b1fb265b35cfdda1f325c276056e1a85bd7166bda569a WatchSource:0}: Error finding container 5c785ed5c92bd8eae23b1fb265b35cfdda1f325c276056e1a85bd7166bda569a: Status 404 returned error can't find the container with id 5c785ed5c92bd8eae23b1fb265b35cfdda1f325c276056e1a85bd7166bda569a Sep 08 16:39:28.949619 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.949582 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 16:39:28.949707 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.949649 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 16:39:28.949707 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:28.949678 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 16:39:29.211558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:29.211470 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" event={"ID":"92f919b7-2340-43d5-8a2f-5d21c06586d8","Type":"ContainerStarted","Data":"eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130"} Sep 08 16:39:29.211558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:29.211521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" event={"ID":"92f919b7-2340-43d5-8a2f-5d21c06586d8","Type":"ContainerStarted","Data":"5c785ed5c92bd8eae23b1fb265b35cfdda1f325c276056e1a85bd7166bda569a"} Sep 08 16:39:29.235291 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:29.235237 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" podStartSLOduration=1.235215832 podStartE2EDuration="1.235215832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:39:29.23456264 +0000 UTC m=+996.705549120" watchObservedRunningTime="2026-09-08 16:39:29.235215832 +0000 UTC m=+996.706202289" Sep 08 16:39:29.664893 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:29.664852 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:30.669963 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:30.669933 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:31.221388 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:31.221361 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:31.222322 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:31.222301 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:33.146947 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.146913 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8"] Sep 08 16:39:33.150568 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.150545 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:33.153084 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.153063 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a4942b-dockercfg-j7gps\"" Sep 08 16:39:33.153203 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.153120 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a4942b\"" Sep 08 16:39:33.170115 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.170091 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz"] Sep 08 16:39:33.179657 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.179635 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8"] Sep 08 16:39:33.179895 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.179874 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:33.180991 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.180961 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz"] Sep 08 16:39:33.226827 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.226799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68254c59-c198-4516-9a85-ded86eff69cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8\" (UID: \"68254c59-c198-4516-9a85-ded86eff69cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:33.226973 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.226835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz58m\" (UniqueName: \"kubernetes.io/projected/68254c59-c198-4516-9a85-ded86eff69cf-kube-api-access-rz58m\") pod \"payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8\" (UID: \"68254c59-c198-4516-9a85-ded86eff69cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:33.328430 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.328392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz\" (UID: \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:33.328626 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.328603 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwlq\" (UniqueName: \"kubernetes.io/projected/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-kube-api-access-mgwlq\") pod \"payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz\" (UID: \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:33.328722 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.328705 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68254c59-c198-4516-9a85-ded86eff69cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8\" (UID: \"68254c59-c198-4516-9a85-ded86eff69cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:33.328788 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.328760 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz58m\" (UniqueName: \"kubernetes.io/projected/68254c59-c198-4516-9a85-ded86eff69cf-kube-api-access-rz58m\") pod \"payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8\" (UID: \"68254c59-c198-4516-9a85-ded86eff69cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:33.329376 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.329356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68254c59-c198-4516-9a85-ded86eff69cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8\" (UID: \"68254c59-c198-4516-9a85-ded86eff69cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:33.338836 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.338815 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz58m\" (UniqueName: \"kubernetes.io/projected/68254c59-c198-4516-9a85-ded86eff69cf-kube-api-access-rz58m\") pod \"payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8\" (UID: \"68254c59-c198-4516-9a85-ded86eff69cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:33.429638 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.429602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwlq\" (UniqueName: \"kubernetes.io/projected/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-kube-api-access-mgwlq\") pod \"payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz\" (UID: \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:33.429797 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.429695 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz\" (UID: \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:33.430390 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.430371 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz\" (UID: \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:33.438286 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.438264 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwlq\" (UniqueName: \"kubernetes.io/projected/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-kube-api-access-mgwlq\") pod \"payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz\" (UID: \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:33.470865 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.470836 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:33.493226 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.493205 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:33.744489 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.744464 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8"] Sep 08 16:39:33.746455 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:39:33.746429 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68254c59_c198_4516_9a85_ded86eff69cf.slice/crio-b8d0de727b55a85c1c26ba5ac3ee86b5fc568ab548d1b851b6c52c56885a4f90 WatchSource:0}: Error finding container b8d0de727b55a85c1c26ba5ac3ee86b5fc568ab548d1b851b6c52c56885a4f90: Status 404 returned error can't find the container with id b8d0de727b55a85c1c26ba5ac3ee86b5fc568ab548d1b851b6c52c56885a4f90 Sep 08 16:39:33.775732 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:33.775709 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz"] Sep 08 16:39:33.779109 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:39:33.779080 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1e6dad_4bbf_405c_a0cf_91418ea4c9af.slice/crio-af8beb6a1fd644545336d2982f48bcc382e0b0d7c5680e8b70a7b7e8b77b693b WatchSource:0}: Error finding container af8beb6a1fd644545336d2982f48bcc382e0b0d7c5680e8b70a7b7e8b77b693b: Status 404 returned error can't find the container with id af8beb6a1fd644545336d2982f48bcc382e0b0d7c5680e8b70a7b7e8b77b693b Sep 08 16:39:34.237914 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:34.237877 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" event={"ID":"68254c59-c198-4516-9a85-ded86eff69cf","Type":"ContainerStarted","Data":"e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819"} Sep 08 16:39:34.238406 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:34.237977 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" event={"ID":"68254c59-c198-4516-9a85-ded86eff69cf","Type":"ContainerStarted","Data":"b8d0de727b55a85c1c26ba5ac3ee86b5fc568ab548d1b851b6c52c56885a4f90"} Sep 08 16:39:34.238406 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:34.238013 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:34.239752 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:34.239723 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" event={"ID":"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af","Type":"ContainerStarted","Data":"6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11"} Sep 08 16:39:34.239904 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:34.239770 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" event={"ID":"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af","Type":"ContainerStarted","Data":"af8beb6a1fd644545336d2982f48bcc382e0b0d7c5680e8b70a7b7e8b77b693b"} Sep 08 16:39:34.239904 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:34.239796 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:34.256804 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:34.256762 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" podStartSLOduration=1.25674934 podStartE2EDuration="1.25674934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:39:34.254646683 +0000 UTC m=+1001.725633178" watchObservedRunningTime="2026-09-08 16:39:34.25674934 +0000 UTC m=+1001.727735827" Sep 08 16:39:34.273469 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:34.273425 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" podStartSLOduration=1.273411459 podStartE2EDuration="1.273411459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:39:34.271598033 +0000 UTC m=+1001.742584492" watchObservedRunningTime="2026-09-08 16:39:34.273411459 +0000 UTC m=+1001.744397947" Sep 08 16:39:37.200541 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:37.200510 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:39:45.247537 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:45.247502 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:39:45.247967 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:45.247629 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:39:47.157741 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:47.157285 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": read tcp 10.134.0.2:36978->10.134.0.62:15021: read: connection reset by peer" start-of-body= Sep 08 16:39:47.157741 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:47.157345 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": read tcp 10.134.0.2:36978->10.134.0.62:15021: read: connection reset by peer" Sep 08 16:39:47.313733 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:47.313647 2567 generic.go:356] "Generic (PLEG): container finished" podID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerID="eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130" exitCode=0 Sep 08 16:39:47.313880 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:47.313723 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" event={"ID":"92f919b7-2340-43d5-8a2f-5d21c06586d8","Type":"ContainerDied","Data":"eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130"} Sep 08 16:39:47.314217 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:47.314201 2567 scope.go:117] "RemoveContainer" containerID="eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130" Sep 08 16:39:47.315258 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:47.315227 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 16:39:47.315329 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:47.315297 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 16:39:47.315368 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:47.315328 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 16:39:48.322956 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:48.322923 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" event={"ID":"92f919b7-2340-43d5-8a2f-5d21c06586d8","Type":"ContainerStarted","Data":"0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18"} Sep 08 16:39:48.665216 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:48.665187 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:48.665393 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:48.665368 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 16:39:48.665453 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:48.665424 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 16:39:49.664908 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:49.664874 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 16:39:49.665318 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:49.664930 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 16:39:50.665621 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:50.665585 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 16:39:50.665976 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:50.665640 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 16:39:51.664933 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:51.664897 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 16:39:51.665254 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:51.664951 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 16:39:52.665514 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:52.665486 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 16:39:52.665878 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:52.665538 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 16:39:53.671313 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:53.671276 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:53.671764 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:53.671739 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:39:53.672523 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:39:53.672501 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:40:02.443895 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.443858 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764"] Sep 08 16:40:02.449209 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.449181 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:02.453279 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.453249 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-823397-dockercfg-vd8pz\"" Sep 08 16:40:02.453451 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.453426 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-823397\"" Sep 08 16:40:02.466560 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.466531 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764"] Sep 08 16:40:02.530143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.530060 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b487a056-bd11-44d7-b0a6-de200fc13fa3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764\" (UID: \"b487a056-bd11-44d7-b0a6-de200fc13fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:02.530333 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.530174 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghd5g\" (UniqueName: \"kubernetes.io/projected/b487a056-bd11-44d7-b0a6-de200fc13fa3-kube-api-access-ghd5g\") pod \"payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764\" (UID: \"b487a056-bd11-44d7-b0a6-de200fc13fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:02.631670 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.631635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b487a056-bd11-44d7-b0a6-de200fc13fa3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764\" (UID: \"b487a056-bd11-44d7-b0a6-de200fc13fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:02.631878 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.631688 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghd5g\" (UniqueName: \"kubernetes.io/projected/b487a056-bd11-44d7-b0a6-de200fc13fa3-kube-api-access-ghd5g\") pod \"payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764\" (UID: \"b487a056-bd11-44d7-b0a6-de200fc13fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:02.632488 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.632464 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b487a056-bd11-44d7-b0a6-de200fc13fa3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764\" (UID: \"b487a056-bd11-44d7-b0a6-de200fc13fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:02.641566 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.641535 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghd5g\" (UniqueName: \"kubernetes.io/projected/b487a056-bd11-44d7-b0a6-de200fc13fa3-kube-api-access-ghd5g\") pod \"payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764\" (UID: \"b487a056-bd11-44d7-b0a6-de200fc13fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:02.762490 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:02.762396 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:03.037939 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:03.037913 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764"] Sep 08 16:40:03.040275 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:40:03.040246 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb487a056_bd11_44d7_b0a6_de200fc13fa3.slice/crio-35a3d1f37389c8944b26ca895daf737c2a11b5b8d5ab78493726aa736312bb5f WatchSource:0}: Error finding container 35a3d1f37389c8944b26ca895daf737c2a11b5b8d5ab78493726aa736312bb5f: Status 404 returned error can't find the container with id 35a3d1f37389c8944b26ca895daf737c2a11b5b8d5ab78493726aa736312bb5f Sep 08 16:40:03.403398 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:03.403362 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" event={"ID":"b487a056-bd11-44d7-b0a6-de200fc13fa3","Type":"ContainerStarted","Data":"716bcd6b4efb84f18369c6bbe3f9e325ed4163725e9a0e4f53cd6e4006fbf6b6"} Sep 08 16:40:03.403611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:03.403455 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" event={"ID":"b487a056-bd11-44d7-b0a6-de200fc13fa3","Type":"ContainerStarted","Data":"35a3d1f37389c8944b26ca895daf737c2a11b5b8d5ab78493726aa736312bb5f"} Sep 08 16:40:03.403611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:03.403484 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:03.423408 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:03.423345 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" podStartSLOduration=1.423324371 podStartE2EDuration="1.423324371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:03.421648197 +0000 UTC m=+1030.892634654" watchObservedRunningTime="2026-09-08 16:40:03.423324371 +0000 UTC m=+1030.894310829" Sep 08 16:40:07.204502 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.204464 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7"] Sep 08 16:40:07.205313 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.205256 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" podUID="61129e89-3ce1-42a6-bf31-1f4e841a4fb0" containerName="payload-pre-processing" containerID="cri-o://35102566b1fc929a3c9663933915d8dbad91e134ca0b7a71f622bf06bb90d4b3" gracePeriod=30 Sep 08 16:40:07.431327 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.431296 2567 generic.go:356] "Generic (PLEG): container finished" podID="61129e89-3ce1-42a6-bf31-1f4e841a4fb0" containerID="35102566b1fc929a3c9663933915d8dbad91e134ca0b7a71f622bf06bb90d4b3" exitCode=0 Sep 08 16:40:07.431518 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.431372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" event={"ID":"61129e89-3ce1-42a6-bf31-1f4e841a4fb0","Type":"ContainerDied","Data":"35102566b1fc929a3c9663933915d8dbad91e134ca0b7a71f622bf06bb90d4b3"} Sep 08 16:40:07.471535 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.471462 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:40:07.585409 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.585372 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-plugins-config-volume\") pod \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\" (UID: \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\") " Sep 08 16:40:07.585760 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.585739 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpgtc\" (UniqueName: \"kubernetes.io/projected/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-kube-api-access-vpgtc\") pod \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\" (UID: \"61129e89-3ce1-42a6-bf31-1f4e841a4fb0\") " Sep 08 16:40:07.586427 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.586393 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61129e89-3ce1-42a6-bf31-1f4e841a4fb0" (UID: "61129e89-3ce1-42a6-bf31-1f4e841a4fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:40:07.595205 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.595156 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-kube-api-access-vpgtc" (OuterVolumeSpecName: "kube-api-access-vpgtc") pod "61129e89-3ce1-42a6-bf31-1f4e841a4fb0" (UID: "61129e89-3ce1-42a6-bf31-1f4e841a4fb0"). InnerVolumeSpecName "kube-api-access-vpgtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:40:07.686686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.686642 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpgtc\" (UniqueName: \"kubernetes.io/projected/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-kube-api-access-vpgtc\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:40:07.686686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:07.686678 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61129e89-3ce1-42a6-bf31-1f4e841a4fb0-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:40:08.438797 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:08.438760 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" event={"ID":"61129e89-3ce1-42a6-bf31-1f4e841a4fb0","Type":"ContainerDied","Data":"f09b7971752143f3e21aa4fcc7fac26ae72251666313a7853c07f7746f0654d6"} Sep 08 16:40:08.439230 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:08.438766 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7" Sep 08 16:40:08.439230 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:08.438849 2567 scope.go:117] "RemoveContainer" containerID="35102566b1fc929a3c9663933915d8dbad91e134ca0b7a71f622bf06bb90d4b3" Sep 08 16:40:08.459981 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:08.459956 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7"] Sep 08 16:40:08.465592 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:08.465558 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a69d4afb-c88868754-kg6z7"] Sep 08 16:40:09.053380 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:09.053351 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61129e89-3ce1-42a6-bf31-1f4e841a4fb0" path="/var/lib/kubelet/pods/61129e89-3ce1-42a6-bf31-1f4e841a4fb0/volumes" Sep 08 16:40:14.412698 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:14.412665 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:40:33.734233 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.734197 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf"] Sep 08 16:40:33.734701 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.734628 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61129e89-3ce1-42a6-bf31-1f4e841a4fb0" containerName="payload-pre-processing" Sep 08 16:40:33.734701 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.734639 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61129e89-3ce1-42a6-bf31-1f4e841a4fb0" containerName="payload-pre-processing" Sep 08 16:40:33.734775 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.734733 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="61129e89-3ce1-42a6-bf31-1f4e841a4fb0" containerName="payload-pre-processing" Sep 08 16:40:33.737981 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.737965 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:33.740522 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.740495 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5a3b57-dockercfg-gj8pv\"" Sep 08 16:40:33.740653 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.740538 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5a3b57\"" Sep 08 16:40:33.752296 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.752271 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf"] Sep 08 16:40:33.829699 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.829661 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d72aa-f548-48b8-9f4a-69554171e524-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf\" (UID: \"499d72aa-f548-48b8-9f4a-69554171e524\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:33.829856 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.829748 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tdjf\" (UniqueName: \"kubernetes.io/projected/499d72aa-f548-48b8-9f4a-69554171e524-kube-api-access-5tdjf\") pod \"payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf\" (UID: \"499d72aa-f548-48b8-9f4a-69554171e524\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:33.930488 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.930450 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tdjf\" (UniqueName: \"kubernetes.io/projected/499d72aa-f548-48b8-9f4a-69554171e524-kube-api-access-5tdjf\") pod \"payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf\" (UID: \"499d72aa-f548-48b8-9f4a-69554171e524\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:33.930668 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.930549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d72aa-f548-48b8-9f4a-69554171e524-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf\" (UID: \"499d72aa-f548-48b8-9f4a-69554171e524\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:33.931310 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.931290 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d72aa-f548-48b8-9f4a-69554171e524-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf\" (UID: \"499d72aa-f548-48b8-9f4a-69554171e524\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:33.941978 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:33.941944 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tdjf\" (UniqueName: \"kubernetes.io/projected/499d72aa-f548-48b8-9f4a-69554171e524-kube-api-access-5tdjf\") pod \"payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf\" (UID: \"499d72aa-f548-48b8-9f4a-69554171e524\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:34.048318 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:34.048222 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:34.358099 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:34.358071 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf"] Sep 08 16:40:34.359575 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:40:34.359550 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499d72aa_f548_48b8_9f4a_69554171e524.slice/crio-0211af449b8c9c15e99e2793f5a2bed9d08d05e0d93297ed88524395447950bf WatchSource:0}: Error finding container 0211af449b8c9c15e99e2793f5a2bed9d08d05e0d93297ed88524395447950bf: Status 404 returned error can't find the container with id 0211af449b8c9c15e99e2793f5a2bed9d08d05e0d93297ed88524395447950bf Sep 08 16:40:34.580348 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:34.580299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" event={"ID":"499d72aa-f548-48b8-9f4a-69554171e524","Type":"ContainerStarted","Data":"ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36"} Sep 08 16:40:34.580530 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:34.580366 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" event={"ID":"499d72aa-f548-48b8-9f4a-69554171e524","Type":"ContainerStarted","Data":"0211af449b8c9c15e99e2793f5a2bed9d08d05e0d93297ed88524395447950bf"} Sep 08 16:40:34.580530 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:34.580474 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:34.600975 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:34.600918 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" podStartSLOduration=1.600901875 podStartE2EDuration="1.600901875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:34.599207426 +0000 UTC m=+1062.070193886" watchObservedRunningTime="2026-09-08 16:40:34.600901875 +0000 UTC m=+1062.071888331" Sep 08 16:40:45.589559 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:45.589476 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:40:53.135762 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:40:53.135728 2567 scope.go:117] "RemoveContainer" containerID="5d2eb331ef1fa9cae30a9d6c4d8a09c48fceca9d0d0681da82c5c6d2b5e8904c" Sep 08 16:41:02.898831 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.898792 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc"] Sep 08 16:41:02.904232 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.904211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:02.907468 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.907443 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b0b77f-dockercfg-s5vjp\"" Sep 08 16:41:02.908790 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.908762 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b0b77f\"" Sep 08 16:41:02.914439 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.914414 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc"] Sep 08 16:41:02.923658 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.923621 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt"] Sep 08 16:41:02.925519 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.925491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhzmn\" (UniqueName: \"kubernetes.io/projected/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-kube-api-access-zhzmn\") pod \"payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc\" (UID: \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:02.925651 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.925530 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc\" (UID: \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:02.928242 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.928220 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:02.943454 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:02.943424 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt"] Sep 08 16:41:03.027048 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.026980 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhzmn\" (UniqueName: \"kubernetes.io/projected/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-kube-api-access-zhzmn\") pod \"payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc\" (UID: \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:03.027216 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.027058 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc\" (UID: \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:03.027216 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.027113 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhck\" (UniqueName: \"kubernetes.io/projected/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-kube-api-access-9mhck\") pod \"payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt\" (UID: \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:03.027216 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.027146 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt\" (UID: \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:03.027772 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.027753 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc\" (UID: \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:03.037998 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.037965 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhzmn\" (UniqueName: \"kubernetes.io/projected/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-kube-api-access-zhzmn\") pod \"payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc\" (UID: \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:03.128139 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.128090 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhck\" (UniqueName: \"kubernetes.io/projected/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-kube-api-access-9mhck\") pod \"payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt\" (UID: \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:03.128343 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.128153 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt\" (UID: \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:03.128823 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.128800 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt\" (UID: \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:03.136996 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.136969 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhck\" (UniqueName: \"kubernetes.io/projected/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-kube-api-access-9mhck\") pod \"payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt\" (UID: \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:03.216459 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.216416 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:03.240350 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.240320 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:03.492347 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.492318 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc"] Sep 08 16:41:03.494132 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:41:03.494102 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71208143_22aa_4e0f_8e92_1bd6bd40ba9b.slice/crio-2b5224fb13a116bd8b01debece4d6d2f4dcb891ecd201b41e18b2683c158146b WatchSource:0}: Error finding container 2b5224fb13a116bd8b01debece4d6d2f4dcb891ecd201b41e18b2683c158146b: Status 404 returned error can't find the container with id 2b5224fb13a116bd8b01debece4d6d2f4dcb891ecd201b41e18b2683c158146b Sep 08 16:41:03.531467 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.531441 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt"] Sep 08 16:41:03.534122 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:41:03.534013 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda3d6cd_8d44_4e1e_9240_6e2b98ba4366.slice/crio-2af60f37abd20fa70c7458474143a7620af418e13100e9195b29b650de741de0 WatchSource:0}: Error finding container 2af60f37abd20fa70c7458474143a7620af418e13100e9195b29b650de741de0: Status 404 returned error can't find the container with id 2af60f37abd20fa70c7458474143a7620af418e13100e9195b29b650de741de0 Sep 08 16:41:03.754383 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.754282 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" event={"ID":"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366","Type":"ContainerStarted","Data":"f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd"} Sep 08 16:41:03.754383 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.754338 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" event={"ID":"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366","Type":"ContainerStarted","Data":"2af60f37abd20fa70c7458474143a7620af418e13100e9195b29b650de741de0"} Sep 08 16:41:03.754648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.754475 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:03.756084 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.756060 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" event={"ID":"71208143-22aa-4e0f-8e92-1bd6bd40ba9b","Type":"ContainerStarted","Data":"7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2"} Sep 08 16:41:03.756205 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.756107 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" event={"ID":"71208143-22aa-4e0f-8e92-1bd6bd40ba9b","Type":"ContainerStarted","Data":"2b5224fb13a116bd8b01debece4d6d2f4dcb891ecd201b41e18b2683c158146b"} Sep 08 16:41:03.756205 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.756137 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:03.774297 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.774247 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" podStartSLOduration=1.774230894 podStartE2EDuration="1.774230894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:03.772476762 +0000 UTC m=+1091.243463243" watchObservedRunningTime="2026-09-08 16:41:03.774230894 +0000 UTC m=+1091.245217349" Sep 08 16:41:03.793180 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:03.793117 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" podStartSLOduration=1.7930971009999999 podStartE2EDuration="1.793097101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:03.791516942 +0000 UTC m=+1091.262503395" watchObservedRunningTime="2026-09-08 16:41:03.793097101 +0000 UTC m=+1091.264083634" Sep 08 16:41:14.764539 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:14.764506 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:41:14.764916 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:14.764574 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:41:45.936071 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:45.936036 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz"] Sep 08 16:41:45.947073 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:45.946810 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:41:45.949594 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:45.949564 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b16e488f\"" Sep 08 16:41:45.949746 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:45.949643 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b16e488f-dockercfg-gtfxw\"" Sep 08 16:41:45.950152 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:45.950125 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz"] Sep 08 16:41:46.052253 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.052218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe31c8dd-dbb1-47f8-be33-7e674214c354-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz\" (UID: \"fe31c8dd-dbb1-47f8-be33-7e674214c354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:41:46.052253 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.052263 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxq8t\" (UniqueName: \"kubernetes.io/projected/fe31c8dd-dbb1-47f8-be33-7e674214c354-kube-api-access-jxq8t\") pod \"payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz\" (UID: \"fe31c8dd-dbb1-47f8-be33-7e674214c354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:41:46.153832 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.153793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe31c8dd-dbb1-47f8-be33-7e674214c354-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz\" (UID: \"fe31c8dd-dbb1-47f8-be33-7e674214c354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:41:46.154042 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.153874 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxq8t\" (UniqueName: \"kubernetes.io/projected/fe31c8dd-dbb1-47f8-be33-7e674214c354-kube-api-access-jxq8t\") pod \"payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz\" (UID: \"fe31c8dd-dbb1-47f8-be33-7e674214c354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:41:46.155145 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.155118 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe31c8dd-dbb1-47f8-be33-7e674214c354-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz\" (UID: \"fe31c8dd-dbb1-47f8-be33-7e674214c354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:41:46.164558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.164522 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxq8t\" (UniqueName: \"kubernetes.io/projected/fe31c8dd-dbb1-47f8-be33-7e674214c354-kube-api-access-jxq8t\") pod \"payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz\" (UID: \"fe31c8dd-dbb1-47f8-be33-7e674214c354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:41:46.259383 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.259292 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:41:46.533364 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.533341 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz"] Sep 08 16:41:46.535077 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:41:46.535053 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe31c8dd_dbb1_47f8_be33_7e674214c354.slice/crio-7286a0794f8ffa81cdd0eafeea78a07df0ae716c9c7d6197d68a7f7ec6021f38 WatchSource:0}: Error finding container 7286a0794f8ffa81cdd0eafeea78a07df0ae716c9c7d6197d68a7f7ec6021f38: Status 404 returned error can't find the container with id 7286a0794f8ffa81cdd0eafeea78a07df0ae716c9c7d6197d68a7f7ec6021f38 Sep 08 16:41:46.999467 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.999418 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" event={"ID":"fe31c8dd-dbb1-47f8-be33-7e674214c354","Type":"ContainerStarted","Data":"b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b"} Sep 08 16:41:46.999910 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.999485 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" event={"ID":"fe31c8dd-dbb1-47f8-be33-7e674214c354","Type":"ContainerStarted","Data":"7286a0794f8ffa81cdd0eafeea78a07df0ae716c9c7d6197d68a7f7ec6021f38"} Sep 08 16:41:46.999910 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:46.999514 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:41:47.028787 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:47.028736 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" podStartSLOduration=2.028719874 podStartE2EDuration="2.028719874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:47.026374104 +0000 UTC m=+1134.497360560" watchObservedRunningTime="2026-09-08 16:41:47.028719874 +0000 UTC m=+1134.499706330" Sep 08 16:41:58.007666 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:41:58.007623 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:42:48.419693 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:48.419615 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5"] Sep 08 16:42:48.423436 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:48.423420 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" Sep 08 16:42:48.425842 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:48.425825 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:42:48.425929 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:48.425871 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzslb\"" Sep 08 16:42:48.430495 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:48.430469 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5"] Sep 08 16:42:48.571903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:48.571864 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01953610-30cf-41b4-9ea0-ed9d6d4f4525-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5\" (UID: \"01953610-30cf-41b4-9ea0-ed9d6d4f4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" Sep 08 16:42:48.672774 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:48.672686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01953610-30cf-41b4-9ea0-ed9d6d4f4525-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5\" (UID: \"01953610-30cf-41b4-9ea0-ed9d6d4f4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" Sep 08 16:42:48.673391 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:48.673369 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01953610-30cf-41b4-9ea0-ed9d6d4f4525-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5\" (UID: \"01953610-30cf-41b4-9ea0-ed9d6d4f4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" Sep 08 16:42:48.735217 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:48.735174 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" Sep 08 16:42:49.009780 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:49.009749 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5"] Sep 08 16:42:49.011812 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:42:49.011788 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01953610_30cf_41b4_9ea0_ed9d6d4f4525.slice/crio-b27542d0b0920199e7d98a5fefda97a0a38f1f14a3320a7a7506fb736b77efdf WatchSource:0}: Error finding container b27542d0b0920199e7d98a5fefda97a0a38f1f14a3320a7a7506fb736b77efdf: Status 404 returned error can't find the container with id b27542d0b0920199e7d98a5fefda97a0a38f1f14a3320a7a7506fb736b77efdf Sep 08 16:42:49.344866 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:49.344827 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" event={"ID":"01953610-30cf-41b4-9ea0-ed9d6d4f4525","Type":"ContainerStarted","Data":"b27542d0b0920199e7d98a5fefda97a0a38f1f14a3320a7a7506fb736b77efdf"} Sep 08 16:42:50.352095 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:50.352061 2567 generic.go:356] "Generic (PLEG): container finished" podID="01953610-30cf-41b4-9ea0-ed9d6d4f4525" containerID="b48fc360be87aca58caa4461472841e1fcfac824ffd13789f20abbd33f0d47d8" exitCode=0 Sep 08 16:42:50.352484 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:50.352131 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" event={"ID":"01953610-30cf-41b4-9ea0-ed9d6d4f4525","Type":"ContainerDied","Data":"b48fc360be87aca58caa4461472841e1fcfac824ffd13789f20abbd33f0d47d8"} Sep 08 16:42:51.497541 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:51.497512 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" Sep 08 16:42:51.598176 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:51.598147 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01953610-30cf-41b4-9ea0-ed9d6d4f4525-maas-api-service-ca\") pod \"01953610-30cf-41b4-9ea0-ed9d6d4f4525\" (UID: \"01953610-30cf-41b4-9ea0-ed9d6d4f4525\") " Sep 08 16:42:51.598476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:51.598451 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01953610-30cf-41b4-9ea0-ed9d6d4f4525-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01953610-30cf-41b4-9ea0-ed9d6d4f4525" (UID: "01953610-30cf-41b4-9ea0-ed9d6d4f4525"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:42:51.699398 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:51.699363 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01953610-30cf-41b4-9ea0-ed9d6d4f4525-maas-api-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:42:52.364181 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:52.364149 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" Sep 08 16:42:52.364181 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:52.364168 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5" event={"ID":"01953610-30cf-41b4-9ea0-ed9d6d4f4525","Type":"ContainerDied","Data":"b27542d0b0920199e7d98a5fefda97a0a38f1f14a3320a7a7506fb736b77efdf"} Sep 08 16:42:52.364402 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:52.364215 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b27542d0b0920199e7d98a5fefda97a0a38f1f14a3320a7a7506fb736b77efdf" Sep 08 16:42:53.032558 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:53.032525 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:42:53.034638 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:42:53.034614 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:43:04.126504 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.126468 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz"] Sep 08 16:43:04.127343 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.126740 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" podUID="fe31c8dd-dbb1-47f8-be33-7e674214c354" containerName="payload-pre-processing" containerID="cri-o://b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b" gracePeriod=30 Sep 08 16:43:04.297630 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.297590 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5"] Sep 08 16:43:04.301786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.301752 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b16e488f-r2lv5"] Sep 08 16:43:04.388139 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.387996 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:43:04.424456 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.424430 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxq8t\" (UniqueName: \"kubernetes.io/projected/fe31c8dd-dbb1-47f8-be33-7e674214c354-kube-api-access-jxq8t\") pod \"fe31c8dd-dbb1-47f8-be33-7e674214c354\" (UID: \"fe31c8dd-dbb1-47f8-be33-7e674214c354\") " Sep 08 16:43:04.424635 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.424531 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe31c8dd-dbb1-47f8-be33-7e674214c354-plugins-config-volume\") pod \"fe31c8dd-dbb1-47f8-be33-7e674214c354\" (UID: \"fe31c8dd-dbb1-47f8-be33-7e674214c354\") " Sep 08 16:43:04.424955 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.424924 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe31c8dd-dbb1-47f8-be33-7e674214c354-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe31c8dd-dbb1-47f8-be33-7e674214c354" (UID: "fe31c8dd-dbb1-47f8-be33-7e674214c354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:43:04.426835 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.426806 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe31c8dd-dbb1-47f8-be33-7e674214c354-kube-api-access-jxq8t" (OuterVolumeSpecName: "kube-api-access-jxq8t") pod "fe31c8dd-dbb1-47f8-be33-7e674214c354" (UID: "fe31c8dd-dbb1-47f8-be33-7e674214c354"). InnerVolumeSpecName "kube-api-access-jxq8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:43:04.430054 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.430030 2567 generic.go:356] "Generic (PLEG): container finished" podID="fe31c8dd-dbb1-47f8-be33-7e674214c354" containerID="b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b" exitCode=0 Sep 08 16:43:04.430158 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.430084 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" Sep 08 16:43:04.430158 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.430127 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" event={"ID":"fe31c8dd-dbb1-47f8-be33-7e674214c354","Type":"ContainerDied","Data":"b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b"} Sep 08 16:43:04.430235 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.430191 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz" event={"ID":"fe31c8dd-dbb1-47f8-be33-7e674214c354","Type":"ContainerDied","Data":"7286a0794f8ffa81cdd0eafeea78a07df0ae716c9c7d6197d68a7f7ec6021f38"} Sep 08 16:43:04.430235 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.430222 2567 scope.go:117] "RemoveContainer" containerID="b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b" Sep 08 16:43:04.443912 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.443886 2567 scope.go:117] "RemoveContainer" containerID="b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b" Sep 08 16:43:04.444441 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:43:04.444413 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b\": container with ID starting with b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b not found: ID does not exist" containerID="b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b" Sep 08 16:43:04.444534 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.444453 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b"} err="failed to get container status \"b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b\": rpc error: code = NotFound desc = could not find container \"b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b\": container with ID starting with b07b06ab9eedc9dda1afa083753854de84fac8dc692951ea04534be122f89f4b not found: ID does not exist" Sep 08 16:43:04.453597 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.453572 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz"] Sep 08 16:43:04.459036 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.459002 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b16e488f-6b675544cd-8f5fz"] Sep 08 16:43:04.525786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.525749 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxq8t\" (UniqueName: \"kubernetes.io/projected/fe31c8dd-dbb1-47f8-be33-7e674214c354-kube-api-access-jxq8t\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:43:04.525786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:04.525778 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe31c8dd-dbb1-47f8-be33-7e674214c354-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:43:05.054048 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:05.054003 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01953610-30cf-41b4-9ea0-ed9d6d4f4525" path="/var/lib/kubelet/pods/01953610-30cf-41b4-9ea0-ed9d6d4f4525/volumes" Sep 08 16:43:05.054410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:05.054398 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe31c8dd-dbb1-47f8-be33-7e674214c354" path="/var/lib/kubelet/pods/fe31c8dd-dbb1-47f8-be33-7e674214c354/volumes" Sep 08 16:43:53.178153 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:53.178125 2567 scope.go:117] "RemoveContainer" containerID="be3f21790998890dbf0805ecf6e6f2ac773eb5971917ed368f2980147208c1e6" Sep 08 16:43:53.186788 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:43:53.186765 2567 scope.go:117] "RemoveContainer" containerID="045aa947b99d6c374285935ba1879c0bae9ac52f4761502547f8892e1261dff6" Sep 08 16:44:53.214514 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:44:53.214486 2567 scope.go:117] "RemoveContainer" containerID="73a543356745ccf979b09be21f74d9ac771913b553c40b8b394d7813e35376d3" Sep 08 16:44:53.223461 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:44:53.223437 2567 scope.go:117] "RemoveContainer" containerID="651dd9a18c3107321261d33a3d24976fea8aa54a8ca1e2feeeb988ee117c50f2" Sep 08 16:45:00.168943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.168910 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj"] Sep 08 16:45:00.169368 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.169355 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe31c8dd-dbb1-47f8-be33-7e674214c354" containerName="payload-pre-processing" Sep 08 16:45:00.169414 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.169370 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe31c8dd-dbb1-47f8-be33-7e674214c354" containerName="payload-pre-processing" Sep 08 16:45:00.169414 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.169395 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01953610-30cf-41b4-9ea0-ed9d6d4f4525" containerName="revoke-keys" Sep 08 16:45:00.169414 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.169401 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="01953610-30cf-41b4-9ea0-ed9d6d4f4525" containerName="revoke-keys" Sep 08 16:45:00.169512 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.169472 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe31c8dd-dbb1-47f8-be33-7e674214c354" containerName="payload-pre-processing" Sep 08 16:45:00.169512 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.169479 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="01953610-30cf-41b4-9ea0-ed9d6d4f4525" containerName="revoke-keys" Sep 08 16:45:00.174271 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.174248 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj" Sep 08 16:45:00.176687 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.176665 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzslb\"" Sep 08 16:45:00.183959 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.183941 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj" Sep 08 16:45:00.192427 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.192399 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj"] Sep 08 16:45:00.194679 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.194650 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw"] Sep 08 16:45:00.199529 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.199498 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp"] Sep 08 16:45:00.199694 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.199675 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw" Sep 08 16:45:00.204470 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.204442 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp" Sep 08 16:45:00.228583 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.226630 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw"] Sep 08 16:45:00.231484 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.230887 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp"] Sep 08 16:45:00.241989 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.241774 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw" Sep 08 16:45:00.249373 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.249167 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp" Sep 08 16:45:00.488386 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.488358 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj"] Sep 08 16:45:00.489648 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:45:00.489618 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7645ad0_f3bf_4a38_afd5_5d968cdd8d9a.slice/crio-13352e7b25c42dd268dc94f0045f8f5c34f7f482f5723d3e19cc7a0202b8e435 WatchSource:0}: Error finding container 13352e7b25c42dd268dc94f0045f8f5c34f7f482f5723d3e19cc7a0202b8e435: Status 404 returned error can't find the container with id 13352e7b25c42dd268dc94f0045f8f5c34f7f482f5723d3e19cc7a0202b8e435 Sep 08 16:45:00.542334 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.542307 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw"] Sep 08 16:45:00.544237 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:45:00.544208 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda872bb96_db55_40ef_a0b6_6e33e8272b76.slice/crio-217b2b458b72672f1830d8dd3353fab25a349170608c260218ecf9b5ccaa81f6 WatchSource:0}: Error finding container 217b2b458b72672f1830d8dd3353fab25a349170608c260218ecf9b5ccaa81f6: Status 404 returned error can't find the container with id 217b2b458b72672f1830d8dd3353fab25a349170608c260218ecf9b5ccaa81f6 Sep 08 16:45:00.548786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:00.548726 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp"] Sep 08 16:45:00.550793 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:45:00.550772 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d52ce4_15c6_48c0_8aef_6db7e85b75bd.slice/crio-bac485bde6c180d64d4010530f5f3d548c6a3de47b1e00eaba316fcf1e8ed68b WatchSource:0}: Error finding container bac485bde6c180d64d4010530f5f3d548c6a3de47b1e00eaba316fcf1e8ed68b: Status 404 returned error can't find the container with id bac485bde6c180d64d4010530f5f3d548c6a3de47b1e00eaba316fcf1e8ed68b Sep 08 16:45:01.082861 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:01.082775 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj" event={"ID":"d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a","Type":"ContainerStarted","Data":"1e06525ff089c461f603784210a69d186266cdbafba355e45c7784a2e56b197c"} Sep 08 16:45:01.082861 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:01.082834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj" event={"ID":"d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a","Type":"ContainerStarted","Data":"13352e7b25c42dd268dc94f0045f8f5c34f7f482f5723d3e19cc7a0202b8e435"} Sep 08 16:45:01.084422 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:01.084394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp" event={"ID":"31d52ce4-15c6-48c0-8aef-6db7e85b75bd","Type":"ContainerStarted","Data":"6bce1a071dd2bd27c1977aa499940e33ab5582bc3dd3231e1535b26f854279d6"} Sep 08 16:45:01.084529 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:01.084443 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp" event={"ID":"31d52ce4-15c6-48c0-8aef-6db7e85b75bd","Type":"ContainerStarted","Data":"bac485bde6c180d64d4010530f5f3d548c6a3de47b1e00eaba316fcf1e8ed68b"} Sep 08 16:45:01.085866 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:01.085835 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw" event={"ID":"a872bb96-db55-40ef-a0b6-6e33e8272b76","Type":"ContainerStarted","Data":"6b5f38ad0a8ce907d033c21cd0313cf8818d1f4f85a90fce3d32ef053f238ead"} Sep 08 16:45:01.085986 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:01.085874 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw" event={"ID":"a872bb96-db55-40ef-a0b6-6e33e8272b76","Type":"ContainerStarted","Data":"217b2b458b72672f1830d8dd3353fab25a349170608c260218ecf9b5ccaa81f6"} Sep 08 16:45:01.104943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:01.104885 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj" podStartSLOduration=1.104868544 podStartE2EDuration="1.104868544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:01.103428382 +0000 UTC m=+1328.574414837" watchObservedRunningTime="2026-09-08 16:45:01.104868544 +0000 UTC m=+1328.575854999" Sep 08 16:45:01.125922 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:01.125866 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp" podStartSLOduration=1.125848937 podStartE2EDuration="1.125848937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:01.124744844 +0000 UTC m=+1328.595731302" watchObservedRunningTime="2026-09-08 16:45:01.125848937 +0000 UTC m=+1328.596835440" Sep 08 16:45:01.145326 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:01.145275 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw" podStartSLOduration=1.145258822 podStartE2EDuration="1.145258822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:01.143741639 +0000 UTC m=+1328.614728094" watchObservedRunningTime="2026-09-08 16:45:01.145258822 +0000 UTC m=+1328.616245278" Sep 08 16:45:02.092763 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:02.092725 2567 generic.go:356] "Generic (PLEG): container finished" podID="31d52ce4-15c6-48c0-8aef-6db7e85b75bd" containerID="6bce1a071dd2bd27c1977aa499940e33ab5582bc3dd3231e1535b26f854279d6" exitCode=0 Sep 08 16:45:02.093316 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:02.092810 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp" event={"ID":"31d52ce4-15c6-48c0-8aef-6db7e85b75bd","Type":"ContainerDied","Data":"6bce1a071dd2bd27c1977aa499940e33ab5582bc3dd3231e1535b26f854279d6"} Sep 08 16:45:02.094458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:02.094438 2567 generic.go:356] "Generic (PLEG): container finished" podID="a872bb96-db55-40ef-a0b6-6e33e8272b76" containerID="6b5f38ad0a8ce907d033c21cd0313cf8818d1f4f85a90fce3d32ef053f238ead" exitCode=0 Sep 08 16:45:02.094561 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:02.094464 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw" event={"ID":"a872bb96-db55-40ef-a0b6-6e33e8272b76","Type":"ContainerDied","Data":"6b5f38ad0a8ce907d033c21cd0313cf8818d1f4f85a90fce3d32ef053f238ead"} Sep 08 16:45:02.096089 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:02.096064 2567 generic.go:356] "Generic (PLEG): container finished" podID="d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a" containerID="1e06525ff089c461f603784210a69d186266cdbafba355e45c7784a2e56b197c" exitCode=0 Sep 08 16:45:02.096200 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:02.096095 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj" event={"ID":"d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a","Type":"ContainerDied","Data":"1e06525ff089c461f603784210a69d186266cdbafba355e45c7784a2e56b197c"} Sep 08 16:45:03.247527 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:03.247497 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj" Sep 08 16:45:03.291635 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:03.291607 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp" Sep 08 16:45:03.329642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:03.329616 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw" Sep 08 16:45:04.109748 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:04.109723 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj" Sep 08 16:45:04.109927 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:04.109763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj" event={"ID":"d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a","Type":"ContainerDied","Data":"13352e7b25c42dd268dc94f0045f8f5c34f7f482f5723d3e19cc7a0202b8e435"} Sep 08 16:45:04.109927 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:04.109810 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13352e7b25c42dd268dc94f0045f8f5c34f7f482f5723d3e19cc7a0202b8e435" Sep 08 16:45:04.111484 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:04.111456 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp" Sep 08 16:45:04.111586 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:04.111482 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp" event={"ID":"31d52ce4-15c6-48c0-8aef-6db7e85b75bd","Type":"ContainerDied","Data":"bac485bde6c180d64d4010530f5f3d548c6a3de47b1e00eaba316fcf1e8ed68b"} Sep 08 16:45:04.111586 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:04.111533 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bac485bde6c180d64d4010530f5f3d548c6a3de47b1e00eaba316fcf1e8ed68b" Sep 08 16:45:04.113014 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:04.112984 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw" Sep 08 16:45:04.113153 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:04.112983 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw" event={"ID":"a872bb96-db55-40ef-a0b6-6e33e8272b76","Type":"ContainerDied","Data":"217b2b458b72672f1830d8dd3353fab25a349170608c260218ecf9b5ccaa81f6"} Sep 08 16:45:04.113153 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:04.113084 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="217b2b458b72672f1830d8dd3353fab25a349170608c260218ecf9b5ccaa81f6" Sep 08 16:45:29.230711 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.230613 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l"] Sep 08 16:45:29.231328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.231186 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a" containerName="cleanup" Sep 08 16:45:29.231328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.231202 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a" containerName="cleanup" Sep 08 16:45:29.231328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.231211 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d52ce4-15c6-48c0-8aef-6db7e85b75bd" containerName="cleanup" Sep 08 16:45:29.231328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.231217 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d52ce4-15c6-48c0-8aef-6db7e85b75bd" containerName="cleanup" Sep 08 16:45:29.231328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.231244 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a872bb96-db55-40ef-a0b6-6e33e8272b76" containerName="cleanup" Sep 08 16:45:29.231328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.231251 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a872bb96-db55-40ef-a0b6-6e33e8272b76" containerName="cleanup" Sep 08 16:45:29.231328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.231315 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a872bb96-db55-40ef-a0b6-6e33e8272b76" containerName="cleanup" Sep 08 16:45:29.231328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.231325 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d52ce4-15c6-48c0-8aef-6db7e85b75bd" containerName="cleanup" Sep 08 16:45:29.231328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.231333 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a" containerName="cleanup" Sep 08 16:45:29.234617 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.234598 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w"] Sep 08 16:45:29.234749 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.234734 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:29.237202 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.237181 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f2ae08\"" Sep 08 16:45:29.238196 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.238183 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:29.240727 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.240710 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f2ae08-dockercfg-dd6wr\"" Sep 08 16:45:29.253010 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.252982 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w"] Sep 08 16:45:29.256127 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.256107 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l"] Sep 08 16:45:29.333732 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.333697 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7rw\" (UniqueName: \"kubernetes.io/projected/16bcf6eb-70a9-4a80-81e2-8073f550a76c-kube-api-access-fz7rw\") pod \"payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l\" (UID: \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:29.333938 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.333744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c0a9f-50a6-4986-b51b-86db6f3e73e1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w\" (UID: \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\") " pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:29.333938 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.333797 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bcf6eb-70a9-4a80-81e2-8073f550a76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l\" (UID: \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:29.333938 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.333853 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kszhx\" (UniqueName: \"kubernetes.io/projected/379c0a9f-50a6-4986-b51b-86db6f3e73e1-kube-api-access-kszhx\") pod \"payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w\" (UID: \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\") " pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:29.434972 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.434936 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7rw\" (UniqueName: \"kubernetes.io/projected/16bcf6eb-70a9-4a80-81e2-8073f550a76c-kube-api-access-fz7rw\") pod \"payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l\" (UID: \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:29.435199 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.434984 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c0a9f-50a6-4986-b51b-86db6f3e73e1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w\" (UID: \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\") " pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:29.435199 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.435044 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bcf6eb-70a9-4a80-81e2-8073f550a76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l\" (UID: \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:29.435199 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.435101 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kszhx\" (UniqueName: \"kubernetes.io/projected/379c0a9f-50a6-4986-b51b-86db6f3e73e1-kube-api-access-kszhx\") pod \"payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w\" (UID: \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\") " pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:29.435795 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.435769 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bcf6eb-70a9-4a80-81e2-8073f550a76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l\" (UID: \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:29.435895 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.435770 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c0a9f-50a6-4986-b51b-86db6f3e73e1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w\" (UID: \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\") " pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:29.443315 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.443286 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7rw\" (UniqueName: \"kubernetes.io/projected/16bcf6eb-70a9-4a80-81e2-8073f550a76c-kube-api-access-fz7rw\") pod \"payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l\" (UID: \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:29.443428 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.443344 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kszhx\" (UniqueName: \"kubernetes.io/projected/379c0a9f-50a6-4986-b51b-86db6f3e73e1-kube-api-access-kszhx\") pod \"payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w\" (UID: \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\") " pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:29.546481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.546395 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:29.555611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.555585 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:29.824697 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.824667 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l"] Sep 08 16:45:29.826625 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:45:29.826598 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bcf6eb_70a9_4a80_81e2_8073f550a76c.slice/crio-83178f25a25811180016dc4ee75589a196deba1a54bb2318bd54e248ae874a43 WatchSource:0}: Error finding container 83178f25a25811180016dc4ee75589a196deba1a54bb2318bd54e248ae874a43: Status 404 returned error can't find the container with id 83178f25a25811180016dc4ee75589a196deba1a54bb2318bd54e248ae874a43 Sep 08 16:45:29.838378 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:29.838357 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w"] Sep 08 16:45:29.841273 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:45:29.841251 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379c0a9f_50a6_4986_b51b_86db6f3e73e1.slice/crio-13adeb8b83aa9d197e2501cbc4408fd796b287dd745ac2da66fbfbb3fb4b3a4b WatchSource:0}: Error finding container 13adeb8b83aa9d197e2501cbc4408fd796b287dd745ac2da66fbfbb3fb4b3a4b: Status 404 returned error can't find the container with id 13adeb8b83aa9d197e2501cbc4408fd796b287dd745ac2da66fbfbb3fb4b3a4b Sep 08 16:45:30.254246 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:30.254209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" event={"ID":"379c0a9f-50a6-4986-b51b-86db6f3e73e1","Type":"ContainerStarted","Data":"a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48"} Sep 08 16:45:30.254881 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:30.254277 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:30.254881 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:30.254304 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" event={"ID":"379c0a9f-50a6-4986-b51b-86db6f3e73e1","Type":"ContainerStarted","Data":"13adeb8b83aa9d197e2501cbc4408fd796b287dd745ac2da66fbfbb3fb4b3a4b"} Sep 08 16:45:30.256091 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:30.256059 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" event={"ID":"16bcf6eb-70a9-4a80-81e2-8073f550a76c","Type":"ContainerStarted","Data":"06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008"} Sep 08 16:45:30.256209 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:30.256120 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" event={"ID":"16bcf6eb-70a9-4a80-81e2-8073f550a76c","Type":"ContainerStarted","Data":"83178f25a25811180016dc4ee75589a196deba1a54bb2318bd54e248ae874a43"} Sep 08 16:45:30.256209 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:30.256152 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:30.273013 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:30.272964 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" podStartSLOduration=1.272948481 podStartE2EDuration="1.272948481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:30.272458152 +0000 UTC m=+1357.743444613" watchObservedRunningTime="2026-09-08 16:45:30.272948481 +0000 UTC m=+1357.743934954" Sep 08 16:45:30.292269 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:30.292215 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" podStartSLOduration=1.292196098 podStartE2EDuration="1.292196098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:30.291201506 +0000 UTC m=+1357.762187961" watchObservedRunningTime="2026-09-08 16:45:30.292196098 +0000 UTC m=+1357.763182559" Sep 08 16:45:41.263261 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:41.263229 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:45:41.263680 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:41.263310 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:45:53.247699 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:45:53.247660 2567 scope.go:117] "RemoveContainer" containerID="19f99a67815095f60fb0f2ffa53b961105e660b4d2adb586176cdedf750197f0" Sep 08 16:46:34.240508 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.240470 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w"] Sep 08 16:46:34.241069 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.240743 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" podUID="379c0a9f-50a6-4986-b51b-86db6f3e73e1" containerName="payload-processing" containerID="cri-o://a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48" gracePeriod=30 Sep 08 16:46:34.246975 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.246936 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l"] Sep 08 16:46:34.247410 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.247362 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" podUID="16bcf6eb-70a9-4a80-81e2-8073f550a76c" containerName="payload-pre-processing" containerID="cri-o://06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008" gracePeriod=30 Sep 08 16:46:34.506685 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.506627 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:46:34.538991 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.538967 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:46:34.633471 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.633441 2567 generic.go:356] "Generic (PLEG): container finished" podID="16bcf6eb-70a9-4a80-81e2-8073f550a76c" containerID="06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008" exitCode=0 Sep 08 16:46:34.633642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.633485 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" Sep 08 16:46:34.633642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.633525 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" event={"ID":"16bcf6eb-70a9-4a80-81e2-8073f550a76c","Type":"ContainerDied","Data":"06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008"} Sep 08 16:46:34.633642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.633576 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l" event={"ID":"16bcf6eb-70a9-4a80-81e2-8073f550a76c","Type":"ContainerDied","Data":"83178f25a25811180016dc4ee75589a196deba1a54bb2318bd54e248ae874a43"} Sep 08 16:46:34.633642 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.633599 2567 scope.go:117] "RemoveContainer" containerID="06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008" Sep 08 16:46:34.634876 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.634858 2567 generic.go:356] "Generic (PLEG): container finished" podID="379c0a9f-50a6-4986-b51b-86db6f3e73e1" containerID="a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48" exitCode=0 Sep 08 16:46:34.634961 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.634883 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" event={"ID":"379c0a9f-50a6-4986-b51b-86db6f3e73e1","Type":"ContainerDied","Data":"a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48"} Sep 08 16:46:34.634961 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.634912 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" Sep 08 16:46:34.635077 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.634919 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w" event={"ID":"379c0a9f-50a6-4986-b51b-86db6f3e73e1","Type":"ContainerDied","Data":"13adeb8b83aa9d197e2501cbc4408fd796b287dd745ac2da66fbfbb3fb4b3a4b"} Sep 08 16:46:34.642761 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.642745 2567 scope.go:117] "RemoveContainer" containerID="06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008" Sep 08 16:46:34.643056 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:46:34.643011 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008\": container with ID starting with 06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008 not found: ID does not exist" containerID="06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008" Sep 08 16:46:34.643112 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.643064 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008"} err="failed to get container status \"06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008\": rpc error: code = NotFound desc = could not find container \"06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008\": container with ID starting with 06889718b1f9dbd238b96a2d3b794dafe8d39621ca7621b403c52a6953cfa008 not found: ID does not exist" Sep 08 16:46:34.643112 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.643079 2567 scope.go:117] "RemoveContainer" containerID="a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48" Sep 08 16:46:34.653526 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.653507 2567 scope.go:117] "RemoveContainer" containerID="a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48" Sep 08 16:46:34.653804 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:46:34.653785 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48\": container with ID starting with a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48 not found: ID does not exist" containerID="a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48" Sep 08 16:46:34.653846 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.653811 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48"} err="failed to get container status \"a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48\": rpc error: code = NotFound desc = could not find container \"a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48\": container with ID starting with a77fffd13d7090bb2faa85607031306cac3c0a55330bb5789a655f33c8addf48 not found: ID does not exist" Sep 08 16:46:34.668336 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.668313 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bcf6eb-70a9-4a80-81e2-8073f550a76c-plugins-config-volume\") pod \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\" (UID: \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\") " Sep 08 16:46:34.668417 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.668367 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz7rw\" (UniqueName: \"kubernetes.io/projected/16bcf6eb-70a9-4a80-81e2-8073f550a76c-kube-api-access-fz7rw\") pod \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\" (UID: \"16bcf6eb-70a9-4a80-81e2-8073f550a76c\") " Sep 08 16:46:34.668417 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.668405 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c0a9f-50a6-4986-b51b-86db6f3e73e1-plugins-config-volume\") pod \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\" (UID: \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\") " Sep 08 16:46:34.668498 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.668428 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kszhx\" (UniqueName: \"kubernetes.io/projected/379c0a9f-50a6-4986-b51b-86db6f3e73e1-kube-api-access-kszhx\") pod \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\" (UID: \"379c0a9f-50a6-4986-b51b-86db6f3e73e1\") " Sep 08 16:46:34.668713 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.668695 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bcf6eb-70a9-4a80-81e2-8073f550a76c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16bcf6eb-70a9-4a80-81e2-8073f550a76c" (UID: "16bcf6eb-70a9-4a80-81e2-8073f550a76c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:46:34.668819 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.668793 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379c0a9f-50a6-4986-b51b-86db6f3e73e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "379c0a9f-50a6-4986-b51b-86db6f3e73e1" (UID: "379c0a9f-50a6-4986-b51b-86db6f3e73e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:46:34.670548 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.670527 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bcf6eb-70a9-4a80-81e2-8073f550a76c-kube-api-access-fz7rw" (OuterVolumeSpecName: "kube-api-access-fz7rw") pod "16bcf6eb-70a9-4a80-81e2-8073f550a76c" (UID: "16bcf6eb-70a9-4a80-81e2-8073f550a76c"). InnerVolumeSpecName "kube-api-access-fz7rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:46:34.670650 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.670632 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379c0a9f-50a6-4986-b51b-86db6f3e73e1-kube-api-access-kszhx" (OuterVolumeSpecName: "kube-api-access-kszhx") pod "379c0a9f-50a6-4986-b51b-86db6f3e73e1" (UID: "379c0a9f-50a6-4986-b51b-86db6f3e73e1"). InnerVolumeSpecName "kube-api-access-kszhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:46:34.769898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.769822 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bcf6eb-70a9-4a80-81e2-8073f550a76c-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:46:34.769898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.769848 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz7rw\" (UniqueName: \"kubernetes.io/projected/16bcf6eb-70a9-4a80-81e2-8073f550a76c-kube-api-access-fz7rw\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:46:34.769898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.769858 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c0a9f-50a6-4986-b51b-86db6f3e73e1-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:46:34.769898 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.769866 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kszhx\" (UniqueName: \"kubernetes.io/projected/379c0a9f-50a6-4986-b51b-86db6f3e73e1-kube-api-access-kszhx\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:46:34.963706 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.963673 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l"] Sep 08 16:46:34.970583 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.970557 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f2ae08-5fbff9ddf8-cbk7l"] Sep 08 16:46:34.987274 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.987249 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w"] Sep 08 16:46:34.991493 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:34.991469 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f2ae08-5b55b95469-ztq7w"] Sep 08 16:46:35.053891 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:35.053824 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16bcf6eb-70a9-4a80-81e2-8073f550a76c" path="/var/lib/kubelet/pods/16bcf6eb-70a9-4a80-81e2-8073f550a76c/volumes" Sep 08 16:46:35.054299 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:35.054285 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="379c0a9f-50a6-4986-b51b-86db6f3e73e1" path="/var/lib/kubelet/pods/379c0a9f-50a6-4986-b51b-86db6f3e73e1/volumes" Sep 08 16:46:51.724951 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:51.724866 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw"] Sep 08 16:46:51.727408 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:51.727383 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5a3b57-29814765-8crzw"] Sep 08 16:46:51.755032 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:51.754985 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf"] Sep 08 16:46:51.755260 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:51.755213 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" podUID="499d72aa-f548-48b8-9f4a-69554171e524" containerName="payload-processing" containerID="cri-o://ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36" gracePeriod=30 Sep 08 16:46:52.042521 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.042496 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:46:52.147501 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.147465 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tdjf\" (UniqueName: \"kubernetes.io/projected/499d72aa-f548-48b8-9f4a-69554171e524-kube-api-access-5tdjf\") pod \"499d72aa-f548-48b8-9f4a-69554171e524\" (UID: \"499d72aa-f548-48b8-9f4a-69554171e524\") " Sep 08 16:46:52.147688 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.147532 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d72aa-f548-48b8-9f4a-69554171e524-plugins-config-volume\") pod \"499d72aa-f548-48b8-9f4a-69554171e524\" (UID: \"499d72aa-f548-48b8-9f4a-69554171e524\") " Sep 08 16:46:52.147945 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.147921 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499d72aa-f548-48b8-9f4a-69554171e524-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "499d72aa-f548-48b8-9f4a-69554171e524" (UID: "499d72aa-f548-48b8-9f4a-69554171e524"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:46:52.149709 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.149683 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499d72aa-f548-48b8-9f4a-69554171e524-kube-api-access-5tdjf" (OuterVolumeSpecName: "kube-api-access-5tdjf") pod "499d72aa-f548-48b8-9f4a-69554171e524" (UID: "499d72aa-f548-48b8-9f4a-69554171e524"). InnerVolumeSpecName "kube-api-access-5tdjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:46:52.248692 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.248661 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d72aa-f548-48b8-9f4a-69554171e524-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:46:52.248692 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.248689 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tdjf\" (UniqueName: \"kubernetes.io/projected/499d72aa-f548-48b8-9f4a-69554171e524-kube-api-access-5tdjf\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:46:52.734750 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.734710 2567 generic.go:356] "Generic (PLEG): container finished" podID="499d72aa-f548-48b8-9f4a-69554171e524" containerID="ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36" exitCode=0 Sep 08 16:46:52.735174 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.734774 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" Sep 08 16:46:52.735174 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.734788 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" event={"ID":"499d72aa-f548-48b8-9f4a-69554171e524","Type":"ContainerDied","Data":"ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36"} Sep 08 16:46:52.735174 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.734874 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf" event={"ID":"499d72aa-f548-48b8-9f4a-69554171e524","Type":"ContainerDied","Data":"0211af449b8c9c15e99e2793f5a2bed9d08d05e0d93297ed88524395447950bf"} Sep 08 16:46:52.735174 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.734901 2567 scope.go:117] "RemoveContainer" containerID="ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36" Sep 08 16:46:52.744185 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.744160 2567 scope.go:117] "RemoveContainer" containerID="ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36" Sep 08 16:46:52.744470 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:46:52.744451 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36\": container with ID starting with ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36 not found: ID does not exist" containerID="ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36" Sep 08 16:46:52.744526 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.744482 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36"} err="failed to get container status \"ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36\": rpc error: code = NotFound desc = could not find container \"ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36\": container with ID starting with ab59f9aefe7e26bf31839f69961e26fe8203202bf974b7305c24bc8367507c36 not found: ID does not exist" Sep 08 16:46:52.756264 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.756239 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf"] Sep 08 16:46:52.760616 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:52.760594 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a3b57-769fcf87d7-xcvsf"] Sep 08 16:46:53.052981 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:53.052901 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="499d72aa-f548-48b8-9f4a-69554171e524" path="/var/lib/kubelet/pods/499d72aa-f548-48b8-9f4a-69554171e524/volumes" Sep 08 16:46:53.053284 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:46:53.053269 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a872bb96-db55-40ef-a0b6-6e33e8272b76" path="/var/lib/kubelet/pods/a872bb96-db55-40ef-a0b6-6e33e8272b76/volumes" Sep 08 16:47:09.847105 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:09.847068 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp"] Sep 08 16:47:09.850215 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:09.850191 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0b77f-29814765-d2tvp"] Sep 08 16:47:09.873507 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:09.873468 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt"] Sep 08 16:47:09.873696 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:09.873666 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" podUID="dda3d6cd-8d44-4e1e-9240-6e2b98ba4366" containerName="payload-processing" containerID="cri-o://f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd" gracePeriod=30 Sep 08 16:47:09.888799 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:09.888764 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc"] Sep 08 16:47:09.892057 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:09.889105 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" podUID="71208143-22aa-4e0f-8e92-1bd6bd40ba9b" containerName="payload-pre-processing" containerID="cri-o://7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2" gracePeriod=30 Sep 08 16:47:10.141154 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.141127 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:47:10.172457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.172394 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:47:10.226270 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.226239 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-plugins-config-volume\") pod \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\" (UID: \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\") " Sep 08 16:47:10.226270 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.226276 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mhck\" (UniqueName: \"kubernetes.io/projected/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-kube-api-access-9mhck\") pod \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\" (UID: \"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366\") " Sep 08 16:47:10.226644 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.226622 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dda3d6cd-8d44-4e1e-9240-6e2b98ba4366" (UID: "dda3d6cd-8d44-4e1e-9240-6e2b98ba4366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:47:10.228474 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.228454 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-kube-api-access-9mhck" (OuterVolumeSpecName: "kube-api-access-9mhck") pod "dda3d6cd-8d44-4e1e-9240-6e2b98ba4366" (UID: "dda3d6cd-8d44-4e1e-9240-6e2b98ba4366"). InnerVolumeSpecName "kube-api-access-9mhck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:47:10.327607 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.327573 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhzmn\" (UniqueName: \"kubernetes.io/projected/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-kube-api-access-zhzmn\") pod \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\" (UID: \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\") " Sep 08 16:47:10.327806 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.327694 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-plugins-config-volume\") pod \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\" (UID: \"71208143-22aa-4e0f-8e92-1bd6bd40ba9b\") " Sep 08 16:47:10.328045 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.327998 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:47:10.328146 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.328050 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhck\" (UniqueName: \"kubernetes.io/projected/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366-kube-api-access-9mhck\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:47:10.328146 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.328099 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71208143-22aa-4e0f-8e92-1bd6bd40ba9b" (UID: "71208143-22aa-4e0f-8e92-1bd6bd40ba9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:47:10.329905 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.329885 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-kube-api-access-zhzmn" (OuterVolumeSpecName: "kube-api-access-zhzmn") pod "71208143-22aa-4e0f-8e92-1bd6bd40ba9b" (UID: "71208143-22aa-4e0f-8e92-1bd6bd40ba9b"). InnerVolumeSpecName "kube-api-access-zhzmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:47:10.428832 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.428798 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhzmn\" (UniqueName: \"kubernetes.io/projected/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-kube-api-access-zhzmn\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:47:10.428832 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.428827 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71208143-22aa-4e0f-8e92-1bd6bd40ba9b-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:47:10.832030 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.831924 2567 generic.go:356] "Generic (PLEG): container finished" podID="71208143-22aa-4e0f-8e92-1bd6bd40ba9b" containerID="7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2" exitCode=0 Sep 08 16:47:10.832030 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.831960 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" event={"ID":"71208143-22aa-4e0f-8e92-1bd6bd40ba9b","Type":"ContainerDied","Data":"7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2"} Sep 08 16:47:10.832030 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.831977 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" Sep 08 16:47:10.832328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.832048 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc" event={"ID":"71208143-22aa-4e0f-8e92-1bd6bd40ba9b","Type":"ContainerDied","Data":"2b5224fb13a116bd8b01debece4d6d2f4dcb891ecd201b41e18b2683c158146b"} Sep 08 16:47:10.832328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.832055 2567 scope.go:117] "RemoveContainer" containerID="7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2" Sep 08 16:47:10.833770 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.833750 2567 generic.go:356] "Generic (PLEG): container finished" podID="dda3d6cd-8d44-4e1e-9240-6e2b98ba4366" containerID="f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd" exitCode=0 Sep 08 16:47:10.833920 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.833798 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" Sep 08 16:47:10.833920 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.833844 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" event={"ID":"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366","Type":"ContainerDied","Data":"f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd"} Sep 08 16:47:10.833920 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.833887 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt" event={"ID":"dda3d6cd-8d44-4e1e-9240-6e2b98ba4366","Type":"ContainerDied","Data":"2af60f37abd20fa70c7458474143a7620af418e13100e9195b29b650de741de0"} Sep 08 16:47:10.841113 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.841096 2567 scope.go:117] "RemoveContainer" containerID="7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2" Sep 08 16:47:10.841435 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:47:10.841416 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2\": container with ID starting with 7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2 not found: ID does not exist" containerID="7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2" Sep 08 16:47:10.841511 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.841445 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2"} err="failed to get container status \"7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2\": rpc error: code = NotFound desc = could not find container \"7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2\": container with ID starting with 7880d127ac29cc515c872cd24e38deb8c7e298123d580df6f93e9177a30500a2 not found: ID does not exist" Sep 08 16:47:10.841511 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.841467 2567 scope.go:117] "RemoveContainer" containerID="f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd" Sep 08 16:47:10.850056 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.850001 2567 scope.go:117] "RemoveContainer" containerID="f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd" Sep 08 16:47:10.850330 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:47:10.850290 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd\": container with ID starting with f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd not found: ID does not exist" containerID="f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd" Sep 08 16:47:10.850330 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.850309 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd"} err="failed to get container status \"f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd\": rpc error: code = NotFound desc = could not find container \"f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd\": container with ID starting with f6275d94d10ee5bf8887e2fc1228a7d0de98d477c1f67a06ccae6917a54347dd not found: ID does not exist" Sep 08 16:47:10.870860 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.870834 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt"] Sep 08 16:47:10.882525 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.882498 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0b77f-84f684d55-ptfpt"] Sep 08 16:47:10.900347 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.900323 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc"] Sep 08 16:47:10.906067 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:10.906042 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0b77f-6bd9859c86-mk4bc"] Sep 08 16:47:11.053288 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:11.053258 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d52ce4-15c6-48c0-8aef-6db7e85b75bd" path="/var/lib/kubelet/pods/31d52ce4-15c6-48c0-8aef-6db7e85b75bd/volumes" Sep 08 16:47:11.053582 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:11.053569 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71208143-22aa-4e0f-8e92-1bd6bd40ba9b" path="/var/lib/kubelet/pods/71208143-22aa-4e0f-8e92-1bd6bd40ba9b/volumes" Sep 08 16:47:11.053884 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:11.053873 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dda3d6cd-8d44-4e1e-9240-6e2b98ba4366" path="/var/lib/kubelet/pods/dda3d6cd-8d44-4e1e-9240-6e2b98ba4366/volumes" Sep 08 16:47:18.956615 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.956576 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn"] Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.956988 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dda3d6cd-8d44-4e1e-9240-6e2b98ba4366" containerName="payload-processing" Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.956999 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dda3d6cd-8d44-4e1e-9240-6e2b98ba4366" containerName="payload-processing" Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957013 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379c0a9f-50a6-4986-b51b-86db6f3e73e1" containerName="payload-processing" Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957031 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="379c0a9f-50a6-4986-b51b-86db6f3e73e1" containerName="payload-processing" Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957042 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71208143-22aa-4e0f-8e92-1bd6bd40ba9b" containerName="payload-pre-processing" Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957048 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="71208143-22aa-4e0f-8e92-1bd6bd40ba9b" containerName="payload-pre-processing" Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957056 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16bcf6eb-70a9-4a80-81e2-8073f550a76c" containerName="payload-pre-processing" Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957061 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="16bcf6eb-70a9-4a80-81e2-8073f550a76c" containerName="payload-pre-processing" Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957068 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="499d72aa-f548-48b8-9f4a-69554171e524" containerName="payload-processing" Sep 08 16:47:18.957126 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957073 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="499d72aa-f548-48b8-9f4a-69554171e524" containerName="payload-processing" Sep 08 16:47:18.957478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957138 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="499d72aa-f548-48b8-9f4a-69554171e524" containerName="payload-processing" Sep 08 16:47:18.957478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957145 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="16bcf6eb-70a9-4a80-81e2-8073f550a76c" containerName="payload-pre-processing" Sep 08 16:47:18.957478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957151 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="379c0a9f-50a6-4986-b51b-86db6f3e73e1" containerName="payload-processing" Sep 08 16:47:18.957478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957158 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dda3d6cd-8d44-4e1e-9240-6e2b98ba4366" containerName="payload-processing" Sep 08 16:47:18.957478 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.957165 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="71208143-22aa-4e0f-8e92-1bd6bd40ba9b" containerName="payload-pre-processing" Sep 08 16:47:18.961790 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.961774 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:18.966711 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.966690 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e30975\"" Sep 08 16:47:18.974081 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.974055 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn"] Sep 08 16:47:18.982154 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.982124 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn"] Sep 08 16:47:18.986584 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.986561 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:18.993038 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:18.989674 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e30975-dockercfg-fcxm5\"" Sep 08 16:47:19.000336 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.000308 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn"] Sep 08 16:47:19.118447 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.118409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gxg9\" (UniqueName: \"kubernetes.io/projected/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-kube-api-access-9gxg9\") pod \"payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn\" (UID: \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:19.118624 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.118462 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn\" (UID: \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:19.118624 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.118490 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da8e5a0-af62-4b02-8287-16d671f15881-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn\" (UID: \"1da8e5a0-af62-4b02-8287-16d671f15881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:19.118720 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.118641 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwbn\" (UniqueName: \"kubernetes.io/projected/1da8e5a0-af62-4b02-8287-16d671f15881-kube-api-access-5lwbn\") pod \"payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn\" (UID: \"1da8e5a0-af62-4b02-8287-16d671f15881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:19.220061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.219937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da8e5a0-af62-4b02-8287-16d671f15881-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn\" (UID: \"1da8e5a0-af62-4b02-8287-16d671f15881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:19.220061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.220051 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwbn\" (UniqueName: \"kubernetes.io/projected/1da8e5a0-af62-4b02-8287-16d671f15881-kube-api-access-5lwbn\") pod \"payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn\" (UID: \"1da8e5a0-af62-4b02-8287-16d671f15881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:19.220267 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.220085 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gxg9\" (UniqueName: \"kubernetes.io/projected/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-kube-api-access-9gxg9\") pod \"payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn\" (UID: \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:19.220267 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.220114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn\" (UID: \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:19.220725 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.220704 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da8e5a0-af62-4b02-8287-16d671f15881-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn\" (UID: \"1da8e5a0-af62-4b02-8287-16d671f15881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:19.220830 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.220809 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn\" (UID: \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:19.229185 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.229153 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwbn\" (UniqueName: \"kubernetes.io/projected/1da8e5a0-af62-4b02-8287-16d671f15881-kube-api-access-5lwbn\") pod \"payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn\" (UID: \"1da8e5a0-af62-4b02-8287-16d671f15881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:19.229288 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.229198 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gxg9\" (UniqueName: \"kubernetes.io/projected/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-kube-api-access-9gxg9\") pod \"payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn\" (UID: \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:19.273201 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.273167 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:19.303536 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.303507 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:19.553082 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.553058 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn"] Sep 08 16:47:19.555480 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:47:19.555452 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1fb490_f1f3_47a2_a6d7_78ed42fa7fba.slice/crio-803a198091c30a0684429825d734cf7361e75e344e71416e355cb1d960206c9f WatchSource:0}: Error finding container 803a198091c30a0684429825d734cf7361e75e344e71416e355cb1d960206c9f: Status 404 returned error can't find the container with id 803a198091c30a0684429825d734cf7361e75e344e71416e355cb1d960206c9f Sep 08 16:47:19.589454 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.589427 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn"] Sep 08 16:47:19.591632 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:47:19.591610 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da8e5a0_af62_4b02_8287_16d671f15881.slice/crio-98c13422d12c2f5484d0c987233638f3576393be063d5204ebd73e2d024c5d7c WatchSource:0}: Error finding container 98c13422d12c2f5484d0c987233638f3576393be063d5204ebd73e2d024c5d7c: Status 404 returned error can't find the container with id 98c13422d12c2f5484d0c987233638f3576393be063d5204ebd73e2d024c5d7c Sep 08 16:47:19.887416 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.887311 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" event={"ID":"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba","Type":"ContainerStarted","Data":"c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b"} Sep 08 16:47:19.887416 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.887375 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" event={"ID":"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba","Type":"ContainerStarted","Data":"803a198091c30a0684429825d734cf7361e75e344e71416e355cb1d960206c9f"} Sep 08 16:47:19.887416 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.887407 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:19.889105 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.889077 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" event={"ID":"1da8e5a0-af62-4b02-8287-16d671f15881","Type":"ContainerStarted","Data":"faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961"} Sep 08 16:47:19.889224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.889121 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" event={"ID":"1da8e5a0-af62-4b02-8287-16d671f15881","Type":"ContainerStarted","Data":"98c13422d12c2f5484d0c987233638f3576393be063d5204ebd73e2d024c5d7c"} Sep 08 16:47:19.889224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.889174 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:19.904309 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.904260 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" podStartSLOduration=1.904245726 podStartE2EDuration="1.904245726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:47:19.902804985 +0000 UTC m=+1467.373791441" watchObservedRunningTime="2026-09-08 16:47:19.904245726 +0000 UTC m=+1467.375232181" Sep 08 16:47:19.921077 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:19.921013 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" podStartSLOduration=1.920999021 podStartE2EDuration="1.920999021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:47:19.919437379 +0000 UTC m=+1467.390423836" watchObservedRunningTime="2026-09-08 16:47:19.920999021 +0000 UTC m=+1467.391985477" Sep 08 16:47:30.895895 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:30.895860 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:47:30.896328 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:30.895945 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:47:47.443812 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:47.443783 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn_1da8e5a0-af62-4b02-8287-16d671f15881/payload-processing/0.log" Sep 08 16:47:53.069720 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:53.069686 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:47:53.081458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:47:53.081428 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:48:53.309611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:53.309575 2567 scope.go:117] "RemoveContainer" containerID="b48fc360be87aca58caa4461472841e1fcfac824ffd13789f20abbd33f0d47d8" Sep 08 16:48:54.148292 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:54.148262 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn_1da8e5a0-af62-4b02-8287-16d671f15881/payload-processing/0.log" Sep 08 16:48:59.541283 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.541242 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2"] Sep 08 16:48:59.546277 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.546260 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:48:59.549319 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.549301 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-804182-dockercfg-zgmcp\"" Sep 08 16:48:59.549513 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.549493 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-804182\"" Sep 08 16:48:59.554665 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.554647 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2"] Sep 08 16:48:59.600965 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.600932 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc"] Sep 08 16:48:59.604504 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.604488 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:48:59.615272 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.615246 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc"] Sep 08 16:48:59.679614 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.679581 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlqmk\" (UniqueName: \"kubernetes.io/projected/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-kube-api-access-vlqmk\") pod \"payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2\" (UID: \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:48:59.679780 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.679675 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8a216c-7287-43a4-ad1b-eee341dad085-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc\" (UID: \"fa8a216c-7287-43a4-ad1b-eee341dad085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:48:59.679780 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.679696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2\" (UID: \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:48:59.679780 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.679715 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8gm\" (UniqueName: \"kubernetes.io/projected/fa8a216c-7287-43a4-ad1b-eee341dad085-kube-api-access-gb8gm\") pod \"payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc\" (UID: \"fa8a216c-7287-43a4-ad1b-eee341dad085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:48:59.780827 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.780792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8gm\" (UniqueName: \"kubernetes.io/projected/fa8a216c-7287-43a4-ad1b-eee341dad085-kube-api-access-gb8gm\") pod \"payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc\" (UID: \"fa8a216c-7287-43a4-ad1b-eee341dad085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:48:59.781037 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.780884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlqmk\" (UniqueName: \"kubernetes.io/projected/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-kube-api-access-vlqmk\") pod \"payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2\" (UID: \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:48:59.781037 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.780989 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8a216c-7287-43a4-ad1b-eee341dad085-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc\" (UID: \"fa8a216c-7287-43a4-ad1b-eee341dad085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:48:59.781165 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.781050 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2\" (UID: \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:48:59.781715 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.781694 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8a216c-7287-43a4-ad1b-eee341dad085-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc\" (UID: \"fa8a216c-7287-43a4-ad1b-eee341dad085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:48:59.781763 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.781711 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2\" (UID: \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:48:59.789514 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.789482 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlqmk\" (UniqueName: \"kubernetes.io/projected/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-kube-api-access-vlqmk\") pod \"payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2\" (UID: \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:48:59.789610 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.789540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8gm\" (UniqueName: \"kubernetes.io/projected/fa8a216c-7287-43a4-ad1b-eee341dad085-kube-api-access-gb8gm\") pod \"payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc\" (UID: \"fa8a216c-7287-43a4-ad1b-eee341dad085\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:48:59.857859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.857764 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:48:59.915462 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:48:59.915411 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:49:00.129726 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.129696 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2"] Sep 08 16:49:00.132221 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:49:00.132189 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6ce431_0fb5_4c93_9b1c_b2fbea00c9d3.slice/crio-adf3fcd632c9304967a68e9059d257aa58882d67742c9596760d5cdbc3a50591 WatchSource:0}: Error finding container adf3fcd632c9304967a68e9059d257aa58882d67742c9596760d5cdbc3a50591: Status 404 returned error can't find the container with id adf3fcd632c9304967a68e9059d257aa58882d67742c9596760d5cdbc3a50591 Sep 08 16:49:00.196750 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.196717 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc"] Sep 08 16:49:00.198690 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:49:00.198659 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8a216c_7287_43a4_ad1b_eee341dad085.slice/crio-b49e21af693bd4cf007fe5ca927c3ae0a4bc7dd933fa32414bc21e14591d0305 WatchSource:0}: Error finding container b49e21af693bd4cf007fe5ca927c3ae0a4bc7dd933fa32414bc21e14591d0305: Status 404 returned error can't find the container with id b49e21af693bd4cf007fe5ca927c3ae0a4bc7dd933fa32414bc21e14591d0305 Sep 08 16:49:00.443940 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.443901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" event={"ID":"fa8a216c-7287-43a4-ad1b-eee341dad085","Type":"ContainerStarted","Data":"89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0"} Sep 08 16:49:00.444170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.443966 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" event={"ID":"fa8a216c-7287-43a4-ad1b-eee341dad085","Type":"ContainerStarted","Data":"b49e21af693bd4cf007fe5ca927c3ae0a4bc7dd933fa32414bc21e14591d0305"} Sep 08 16:49:00.444170 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.443998 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:49:00.445573 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.445549 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" event={"ID":"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3","Type":"ContainerStarted","Data":"de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9"} Sep 08 16:49:00.445698 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.445586 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" event={"ID":"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3","Type":"ContainerStarted","Data":"adf3fcd632c9304967a68e9059d257aa58882d67742c9596760d5cdbc3a50591"} Sep 08 16:49:00.445698 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.445679 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:49:00.459963 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.459923 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" podStartSLOduration=1.459908901 podStartE2EDuration="1.459908901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:49:00.458552852 +0000 UTC m=+1567.929539320" watchObservedRunningTime="2026-09-08 16:49:00.459908901 +0000 UTC m=+1567.930895399" Sep 08 16:49:00.476781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:00.476737 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" podStartSLOduration=1.476723246 podStartE2EDuration="1.476723246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:49:00.47432108 +0000 UTC m=+1567.945307537" watchObservedRunningTime="2026-09-08 16:49:00.476723246 +0000 UTC m=+1567.947709702" Sep 08 16:49:11.454175 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:11.454130 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:49:11.454592 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:11.454427 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:49:29.515640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.515601 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2"] Sep 08 16:49:29.516189 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.515880 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" podUID="4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3" containerName="payload-processing" containerID="cri-o://de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9" gracePeriod=30 Sep 08 16:49:29.526246 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.526219 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc"] Sep 08 16:49:29.526501 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.526468 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" podUID="fa8a216c-7287-43a4-ad1b-eee341dad085" containerName="payload-pre-processing" containerID="cri-o://89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0" gracePeriod=30 Sep 08 16:49:29.808446 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.808420 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:49:29.811228 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.811206 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:49:29.885648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.885608 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8a216c-7287-43a4-ad1b-eee341dad085-plugins-config-volume\") pod \"fa8a216c-7287-43a4-ad1b-eee341dad085\" (UID: \"fa8a216c-7287-43a4-ad1b-eee341dad085\") " Sep 08 16:49:29.885648 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.885653 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-plugins-config-volume\") pod \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\" (UID: \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\") " Sep 08 16:49:29.885901 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.885789 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb8gm\" (UniqueName: \"kubernetes.io/projected/fa8a216c-7287-43a4-ad1b-eee341dad085-kube-api-access-gb8gm\") pod \"fa8a216c-7287-43a4-ad1b-eee341dad085\" (UID: \"fa8a216c-7287-43a4-ad1b-eee341dad085\") " Sep 08 16:49:29.885901 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.885814 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlqmk\" (UniqueName: \"kubernetes.io/projected/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-kube-api-access-vlqmk\") pod \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\" (UID: \"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3\") " Sep 08 16:49:29.886044 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.885987 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8a216c-7287-43a4-ad1b-eee341dad085-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa8a216c-7287-43a4-ad1b-eee341dad085" (UID: "fa8a216c-7287-43a4-ad1b-eee341dad085"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:49:29.886115 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.886070 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3" (UID: "4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:49:29.886238 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.886222 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8a216c-7287-43a4-ad1b-eee341dad085-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:49:29.886288 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.886240 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:49:29.888134 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.888112 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-kube-api-access-vlqmk" (OuterVolumeSpecName: "kube-api-access-vlqmk") pod "4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3" (UID: "4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3"). InnerVolumeSpecName "kube-api-access-vlqmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:49:29.888219 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.888132 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8a216c-7287-43a4-ad1b-eee341dad085-kube-api-access-gb8gm" (OuterVolumeSpecName: "kube-api-access-gb8gm") pod "fa8a216c-7287-43a4-ad1b-eee341dad085" (UID: "fa8a216c-7287-43a4-ad1b-eee341dad085"). InnerVolumeSpecName "kube-api-access-gb8gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:49:29.986756 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.986716 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb8gm\" (UniqueName: \"kubernetes.io/projected/fa8a216c-7287-43a4-ad1b-eee341dad085-kube-api-access-gb8gm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:49:29.986756 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:29.986749 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlqmk\" (UniqueName: \"kubernetes.io/projected/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3-kube-api-access-vlqmk\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:49:30.623408 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.623377 2567 generic.go:356] "Generic (PLEG): container finished" podID="fa8a216c-7287-43a4-ad1b-eee341dad085" containerID="89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0" exitCode=0 Sep 08 16:49:30.623809 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.623465 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" Sep 08 16:49:30.623809 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.623462 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" event={"ID":"fa8a216c-7287-43a4-ad1b-eee341dad085","Type":"ContainerDied","Data":"89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0"} Sep 08 16:49:30.623809 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.623642 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc" event={"ID":"fa8a216c-7287-43a4-ad1b-eee341dad085","Type":"ContainerDied","Data":"b49e21af693bd4cf007fe5ca927c3ae0a4bc7dd933fa32414bc21e14591d0305"} Sep 08 16:49:30.623809 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.623683 2567 scope.go:117] "RemoveContainer" containerID="89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0" Sep 08 16:49:30.625158 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.624987 2567 generic.go:356] "Generic (PLEG): container finished" podID="4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3" containerID="de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9" exitCode=0 Sep 08 16:49:30.625158 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.625048 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" Sep 08 16:49:30.625158 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.625062 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" event={"ID":"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3","Type":"ContainerDied","Data":"de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9"} Sep 08 16:49:30.625158 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.625105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2" event={"ID":"4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3","Type":"ContainerDied","Data":"adf3fcd632c9304967a68e9059d257aa58882d67742c9596760d5cdbc3a50591"} Sep 08 16:49:30.633449 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.633410 2567 scope.go:117] "RemoveContainer" containerID="89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0" Sep 08 16:49:30.633732 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:49:30.633714 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0\": container with ID starting with 89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0 not found: ID does not exist" containerID="89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0" Sep 08 16:49:30.633806 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.633742 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0"} err="failed to get container status \"89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0\": rpc error: code = NotFound desc = could not find container \"89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0\": container with ID starting with 89956e02323930569c625864cc9c8403bf15830634e845450e0be4c94b0e7fa0 not found: ID does not exist" Sep 08 16:49:30.633806 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.633760 2567 scope.go:117] "RemoveContainer" containerID="de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9" Sep 08 16:49:30.642442 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.642423 2567 scope.go:117] "RemoveContainer" containerID="de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9" Sep 08 16:49:30.642707 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:49:30.642688 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9\": container with ID starting with de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9 not found: ID does not exist" containerID="de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9" Sep 08 16:49:30.642752 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.642715 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9"} err="failed to get container status \"de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9\": rpc error: code = NotFound desc = could not find container \"de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9\": container with ID starting with de99e389a3110914e5d9a6087efac436d1197f6cc4a9e49c684e0767549a07a9 not found: ID does not exist" Sep 08 16:49:30.651765 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.651741 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc"] Sep 08 16:49:30.653763 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.653744 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-804182-fb467896b-qstrc"] Sep 08 16:49:30.663769 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.663744 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2"] Sep 08 16:49:30.669463 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:30.669440 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-804182-6cddfdd757-8s9z2"] Sep 08 16:49:31.055291 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:31.055253 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3" path="/var/lib/kubelet/pods/4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3/volumes" Sep 08 16:49:31.055786 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:31.055770 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa8a216c-7287-43a4-ad1b-eee341dad085" path="/var/lib/kubelet/pods/fa8a216c-7287-43a4-ad1b-eee341dad085/volumes" Sep 08 16:49:40.093966 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.093924 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5"] Sep 08 16:49:40.095650 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.095623 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8a216c-7287-43a4-ad1b-eee341dad085" containerName="payload-pre-processing" Sep 08 16:49:40.095814 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.095802 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8a216c-7287-43a4-ad1b-eee341dad085" containerName="payload-pre-processing" Sep 08 16:49:40.095946 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.095933 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3" containerName="payload-processing" Sep 08 16:49:40.096061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.096012 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3" containerName="payload-processing" Sep 08 16:49:40.096367 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.096354 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8a216c-7287-43a4-ad1b-eee341dad085" containerName="payload-pre-processing" Sep 08 16:49:40.096463 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.096453 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6ce431-0fb5-4c93-9b1c-b2fbea00c9d3" containerName="payload-processing" Sep 08 16:49:40.104334 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.104305 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5"] Sep 08 16:49:40.104481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.104406 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" Sep 08 16:49:40.107133 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.107111 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:49:40.107344 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.107198 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzslb\"" Sep 08 16:49:40.183990 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.183944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d81a3-7714-4c07-ba98-7fde94dcbb1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5\" (UID: \"362d81a3-7714-4c07-ba98-7fde94dcbb1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" Sep 08 16:49:40.285195 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.285159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d81a3-7714-4c07-ba98-7fde94dcbb1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5\" (UID: \"362d81a3-7714-4c07-ba98-7fde94dcbb1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" Sep 08 16:49:40.285768 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.285740 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d81a3-7714-4c07-ba98-7fde94dcbb1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5\" (UID: \"362d81a3-7714-4c07-ba98-7fde94dcbb1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" Sep 08 16:49:40.417583 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.417546 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" Sep 08 16:49:40.714838 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:40.714809 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5"] Sep 08 16:49:40.716852 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:49:40.716826 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362d81a3_7714_4c07_ba98_7fde94dcbb1c.slice/crio-5304110a7b2ed976304f4fb9182dcd35dcd6f357e11807d9def10e9e358ebe12 WatchSource:0}: Error finding container 5304110a7b2ed976304f4fb9182dcd35dcd6f357e11807d9def10e9e358ebe12: Status 404 returned error can't find the container with id 5304110a7b2ed976304f4fb9182dcd35dcd6f357e11807d9def10e9e358ebe12 Sep 08 16:49:41.689209 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:41.689177 2567 generic.go:356] "Generic (PLEG): container finished" podID="362d81a3-7714-4c07-ba98-7fde94dcbb1c" containerID="7bbb489914725856e6bbda47e304b679e136a36d6ab7cf16cecaf0c2120ee7dc" exitCode=0 Sep 08 16:49:41.689595 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:41.689266 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" event={"ID":"362d81a3-7714-4c07-ba98-7fde94dcbb1c","Type":"ContainerDied","Data":"7bbb489914725856e6bbda47e304b679e136a36d6ab7cf16cecaf0c2120ee7dc"} Sep 08 16:49:41.689595 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:41.689325 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" event={"ID":"362d81a3-7714-4c07-ba98-7fde94dcbb1c","Type":"ContainerStarted","Data":"5304110a7b2ed976304f4fb9182dcd35dcd6f357e11807d9def10e9e358ebe12"} Sep 08 16:49:42.833536 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:42.833506 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" Sep 08 16:49:42.911780 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:42.911742 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d81a3-7714-4c07-ba98-7fde94dcbb1c-maas-api-service-ca\") pod \"362d81a3-7714-4c07-ba98-7fde94dcbb1c\" (UID: \"362d81a3-7714-4c07-ba98-7fde94dcbb1c\") " Sep 08 16:49:42.912108 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:42.912088 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362d81a3-7714-4c07-ba98-7fde94dcbb1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "362d81a3-7714-4c07-ba98-7fde94dcbb1c" (UID: "362d81a3-7714-4c07-ba98-7fde94dcbb1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:49:43.013360 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:43.013272 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d81a3-7714-4c07-ba98-7fde94dcbb1c-maas-api-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:49:43.700442 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:43.700412 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" Sep 08 16:49:43.700611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:43.700407 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5" event={"ID":"362d81a3-7714-4c07-ba98-7fde94dcbb1c","Type":"ContainerDied","Data":"5304110a7b2ed976304f4fb9182dcd35dcd6f357e11807d9def10e9e358ebe12"} Sep 08 16:49:43.700611 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:43.700540 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5304110a7b2ed976304f4fb9182dcd35dcd6f357e11807d9def10e9e358ebe12" Sep 08 16:49:56.341137 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:56.341102 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5"] Sep 08 16:49:56.344524 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:56.344498 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55a703-6f6g5"] Sep 08 16:49:57.053561 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:49:57.053530 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="362d81a3-7714-4c07-ba98-7fde94dcbb1c" path="/var/lib/kubelet/pods/362d81a3-7714-4c07-ba98-7fde94dcbb1c/volumes" Sep 08 16:50:56.499084 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.499050 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn"] Sep 08 16:50:56.499613 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.499264 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" podUID="1da8e5a0-af62-4b02-8287-16d671f15881" containerName="payload-processing" containerID="cri-o://faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961" gracePeriod=30 Sep 08 16:50:56.508940 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.508902 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn"] Sep 08 16:50:56.509432 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.509363 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" podUID="2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba" containerName="payload-pre-processing" containerID="cri-o://c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b" gracePeriod=30 Sep 08 16:50:56.781234 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.781205 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:50:56.806720 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.806698 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:50:56.927059 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.927001 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gxg9\" (UniqueName: \"kubernetes.io/projected/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-kube-api-access-9gxg9\") pod \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\" (UID: \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\") " Sep 08 16:50:56.927227 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.927100 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da8e5a0-af62-4b02-8287-16d671f15881-plugins-config-volume\") pod \"1da8e5a0-af62-4b02-8287-16d671f15881\" (UID: \"1da8e5a0-af62-4b02-8287-16d671f15881\") " Sep 08 16:50:56.927227 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.927127 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-plugins-config-volume\") pod \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\" (UID: \"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba\") " Sep 08 16:50:56.927227 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.927154 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lwbn\" (UniqueName: \"kubernetes.io/projected/1da8e5a0-af62-4b02-8287-16d671f15881-kube-api-access-5lwbn\") pod \"1da8e5a0-af62-4b02-8287-16d671f15881\" (UID: \"1da8e5a0-af62-4b02-8287-16d671f15881\") " Sep 08 16:50:56.927502 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.927473 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba" (UID: "2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:50:56.927573 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.927521 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da8e5a0-af62-4b02-8287-16d671f15881-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1da8e5a0-af62-4b02-8287-16d671f15881" (UID: "1da8e5a0-af62-4b02-8287-16d671f15881"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:50:56.929490 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.929472 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-kube-api-access-9gxg9" (OuterVolumeSpecName: "kube-api-access-9gxg9") pod "2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba" (UID: "2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba"). InnerVolumeSpecName "kube-api-access-9gxg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:50:56.929556 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:56.929524 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da8e5a0-af62-4b02-8287-16d671f15881-kube-api-access-5lwbn" (OuterVolumeSpecName: "kube-api-access-5lwbn") pod "1da8e5a0-af62-4b02-8287-16d671f15881" (UID: "1da8e5a0-af62-4b02-8287-16d671f15881"). InnerVolumeSpecName "kube-api-access-5lwbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:50:57.028423 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.028333 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da8e5a0-af62-4b02-8287-16d671f15881-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:50:57.028423 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.028379 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:50:57.028423 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.028389 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lwbn\" (UniqueName: \"kubernetes.io/projected/1da8e5a0-af62-4b02-8287-16d671f15881-kube-api-access-5lwbn\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:50:57.028423 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.028398 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gxg9\" (UniqueName: \"kubernetes.io/projected/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba-kube-api-access-9gxg9\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:50:57.095821 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.095793 2567 generic.go:356] "Generic (PLEG): container finished" podID="1da8e5a0-af62-4b02-8287-16d671f15881" containerID="faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961" exitCode=0 Sep 08 16:50:57.095969 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.095836 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" Sep 08 16:50:57.095969 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.095887 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" event={"ID":"1da8e5a0-af62-4b02-8287-16d671f15881","Type":"ContainerDied","Data":"faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961"} Sep 08 16:50:57.095969 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.095943 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn" event={"ID":"1da8e5a0-af62-4b02-8287-16d671f15881","Type":"ContainerDied","Data":"98c13422d12c2f5484d0c987233638f3576393be063d5204ebd73e2d024c5d7c"} Sep 08 16:50:57.096196 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.095972 2567 scope.go:117] "RemoveContainer" containerID="faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961" Sep 08 16:50:57.099752 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.097436 2567 generic.go:356] "Generic (PLEG): container finished" podID="2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba" containerID="c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b" exitCode=0 Sep 08 16:50:57.099752 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.097470 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" event={"ID":"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba","Type":"ContainerDied","Data":"c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b"} Sep 08 16:50:57.099752 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.097509 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" event={"ID":"2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba","Type":"ContainerDied","Data":"803a198091c30a0684429825d734cf7361e75e344e71416e355cb1d960206c9f"} Sep 08 16:50:57.099752 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.097567 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn" Sep 08 16:50:57.107173 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.107154 2567 scope.go:117] "RemoveContainer" containerID="faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961" Sep 08 16:50:57.107467 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:50:57.107449 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961\": container with ID starting with faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961 not found: ID does not exist" containerID="faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961" Sep 08 16:50:57.107512 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.107476 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961"} err="failed to get container status \"faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961\": rpc error: code = NotFound desc = could not find container \"faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961\": container with ID starting with faa195fe4bb9c58462b881e47b8379970342c2f1372b9fd6a52814dd3ff3f961 not found: ID does not exist" Sep 08 16:50:57.107512 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.107493 2567 scope.go:117] "RemoveContainer" containerID="c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b" Sep 08 16:50:57.116099 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.116071 2567 scope.go:117] "RemoveContainer" containerID="c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b" Sep 08 16:50:57.116403 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:50:57.116385 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b\": container with ID starting with c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b not found: ID does not exist" containerID="c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b" Sep 08 16:50:57.116458 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.116410 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b"} err="failed to get container status \"c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b\": rpc error: code = NotFound desc = could not find container \"c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b\": container with ID starting with c23abe5828dc34abd3f9ba35466fb2b2e762984bf96ff243190cd5185ce6d43b not found: ID does not exist" Sep 08 16:50:57.118030 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.118005 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn"] Sep 08 16:50:57.124466 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.124445 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30975-84cb54bb4b-x8rxn"] Sep 08 16:50:57.137638 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.137609 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn"] Sep 08 16:50:57.143288 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:57.143262 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30975-5c9ffb87f7-vxxgn"] Sep 08 16:50:59.053143 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:59.053107 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da8e5a0-af62-4b02-8287-16d671f15881" path="/var/lib/kubelet/pods/1da8e5a0-af62-4b02-8287-16d671f15881/volumes" Sep 08 16:50:59.053521 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:50:59.053463 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba" path="/var/lib/kubelet/pods/2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba/volumes" Sep 08 16:51:14.224643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.224606 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz"] Sep 08 16:51:14.225102 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.224916 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" podUID="2a1e6dad-4bbf-405c-a0cf-91418ea4c9af" containerName="payload-processing" containerID="cri-o://6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11" gracePeriod=30 Sep 08 16:51:14.229101 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.229073 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8"] Sep 08 16:51:14.231141 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.230034 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" podUID="68254c59-c198-4516-9a85-ded86eff69cf" containerName="payload-pre-processing" containerID="cri-o://e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819" gracePeriod=30 Sep 08 16:51:14.502943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.502921 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:51:14.505728 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.505710 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:51:14.596752 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.596715 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgwlq\" (UniqueName: \"kubernetes.io/projected/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-kube-api-access-mgwlq\") pod \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\" (UID: \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\") " Sep 08 16:51:14.596949 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.596780 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz58m\" (UniqueName: \"kubernetes.io/projected/68254c59-c198-4516-9a85-ded86eff69cf-kube-api-access-rz58m\") pod \"68254c59-c198-4516-9a85-ded86eff69cf\" (UID: \"68254c59-c198-4516-9a85-ded86eff69cf\") " Sep 08 16:51:14.596949 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.596818 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-plugins-config-volume\") pod \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\" (UID: \"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af\") " Sep 08 16:51:14.596949 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.596855 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68254c59-c198-4516-9a85-ded86eff69cf-plugins-config-volume\") pod \"68254c59-c198-4516-9a85-ded86eff69cf\" (UID: \"68254c59-c198-4516-9a85-ded86eff69cf\") " Sep 08 16:51:14.597415 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.597354 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a1e6dad-4bbf-405c-a0cf-91418ea4c9af" (UID: "2a1e6dad-4bbf-405c-a0cf-91418ea4c9af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:51:14.597698 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.597460 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68254c59-c198-4516-9a85-ded86eff69cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68254c59-c198-4516-9a85-ded86eff69cf" (UID: "68254c59-c198-4516-9a85-ded86eff69cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:51:14.599156 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.599129 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-kube-api-access-mgwlq" (OuterVolumeSpecName: "kube-api-access-mgwlq") pod "2a1e6dad-4bbf-405c-a0cf-91418ea4c9af" (UID: "2a1e6dad-4bbf-405c-a0cf-91418ea4c9af"). InnerVolumeSpecName "kube-api-access-mgwlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:51:14.599244 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.599197 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68254c59-c198-4516-9a85-ded86eff69cf-kube-api-access-rz58m" (OuterVolumeSpecName: "kube-api-access-rz58m") pod "68254c59-c198-4516-9a85-ded86eff69cf" (UID: "68254c59-c198-4516-9a85-ded86eff69cf"). InnerVolumeSpecName "kube-api-access-rz58m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:51:14.698557 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.698528 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwlq\" (UniqueName: \"kubernetes.io/projected/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-kube-api-access-mgwlq\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:14.698557 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.698555 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz58m\" (UniqueName: \"kubernetes.io/projected/68254c59-c198-4516-9a85-ded86eff69cf-kube-api-access-rz58m\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:14.698722 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.698564 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:14.698722 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:14.698575 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68254c59-c198-4516-9a85-ded86eff69cf-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:15.210013 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.209975 2567 generic.go:356] "Generic (PLEG): container finished" podID="2a1e6dad-4bbf-405c-a0cf-91418ea4c9af" containerID="6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11" exitCode=0 Sep 08 16:51:15.210013 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.210010 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" event={"ID":"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af","Type":"ContainerDied","Data":"6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11"} Sep 08 16:51:15.210256 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.210082 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" event={"ID":"2a1e6dad-4bbf-405c-a0cf-91418ea4c9af","Type":"ContainerDied","Data":"af8beb6a1fd644545336d2982f48bcc382e0b0d7c5680e8b70a7b7e8b77b693b"} Sep 08 16:51:15.210256 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.210087 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz" Sep 08 16:51:15.210256 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.210108 2567 scope.go:117] "RemoveContainer" containerID="6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11" Sep 08 16:51:15.211793 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.211754 2567 generic.go:356] "Generic (PLEG): container finished" podID="68254c59-c198-4516-9a85-ded86eff69cf" containerID="e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819" exitCode=0 Sep 08 16:51:15.211906 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.211802 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" event={"ID":"68254c59-c198-4516-9a85-ded86eff69cf","Type":"ContainerDied","Data":"e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819"} Sep 08 16:51:15.211906 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.211822 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" Sep 08 16:51:15.211906 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.211831 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8" event={"ID":"68254c59-c198-4516-9a85-ded86eff69cf","Type":"ContainerDied","Data":"b8d0de727b55a85c1c26ba5ac3ee86b5fc568ab548d1b851b6c52c56885a4f90"} Sep 08 16:51:15.219870 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.219855 2567 scope.go:117] "RemoveContainer" containerID="6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11" Sep 08 16:51:15.220166 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:51:15.220139 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11\": container with ID starting with 6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11 not found: ID does not exist" containerID="6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11" Sep 08 16:51:15.220217 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.220172 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11"} err="failed to get container status \"6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11\": rpc error: code = NotFound desc = could not find container \"6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11\": container with ID starting with 6407156bcbb3d84837d733482b233780090410db1f652fd5d0699bed52d25b11 not found: ID does not exist" Sep 08 16:51:15.220217 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.220196 2567 scope.go:117] "RemoveContainer" containerID="e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819" Sep 08 16:51:15.228488 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.228323 2567 scope.go:117] "RemoveContainer" containerID="e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819" Sep 08 16:51:15.228694 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:51:15.228591 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819\": container with ID starting with e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819 not found: ID does not exist" containerID="e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819" Sep 08 16:51:15.228694 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.228619 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819"} err="failed to get container status \"e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819\": rpc error: code = NotFound desc = could not find container \"e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819\": container with ID starting with e4bf668e09d4ffa3b08fa7f3ed7971190ba76e9affaea8f1823af7f61d202819 not found: ID does not exist" Sep 08 16:51:15.229659 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.229641 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8"] Sep 08 16:51:15.233464 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.233445 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4942b-75d6cd9dcd-2s6b8"] Sep 08 16:51:15.247861 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.247838 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz"] Sep 08 16:51:15.254539 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:15.254520 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4942b-779bc57c4c-vjknz"] Sep 08 16:51:17.053453 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:17.053420 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1e6dad-4bbf-405c-a0cf-91418ea4c9af" path="/var/lib/kubelet/pods/2a1e6dad-4bbf-405c-a0cf-91418ea4c9af/volumes" Sep 08 16:51:17.053903 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:17.053887 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68254c59-c198-4516-9a85-ded86eff69cf" path="/var/lib/kubelet/pods/68254c59-c198-4516-9a85-ded86eff69cf/volumes" Sep 08 16:51:21.213781 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:21.213748 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt"] Sep 08 16:51:21.214245 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:21.213972 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" containerID="cri-o://0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18" gracePeriod=30 Sep 08 16:51:23.674467 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:23.674433 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 16:51:23.674930 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:23.674504 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 16:51:26.452258 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.452232 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:51:26.507837 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.507759 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-envoy\") pod \"92f919b7-2340-43d5-8a2f-5d21c06586d8\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " Sep 08 16:51:26.507837 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.507813 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-podinfo\") pod \"92f919b7-2340-43d5-8a2f-5d21c06586d8\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " Sep 08 16:51:26.508080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.507844 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f919b7-2340-43d5-8a2f-5d21c06586d8-istiod-ca-cert\") pod \"92f919b7-2340-43d5-8a2f-5d21c06586d8\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " Sep 08 16:51:26.508080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.507861 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-credential-socket\") pod \"92f919b7-2340-43d5-8a2f-5d21c06586d8\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " Sep 08 16:51:26.508080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.507893 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-token\") pod \"92f919b7-2340-43d5-8a2f-5d21c06586d8\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " Sep 08 16:51:26.508080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.507919 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhdlr\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-kube-api-access-jhdlr\") pod \"92f919b7-2340-43d5-8a2f-5d21c06586d8\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " Sep 08 16:51:26.508080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.507971 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-data\") pod \"92f919b7-2340-43d5-8a2f-5d21c06586d8\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " Sep 08 16:51:26.508080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508011 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-socket\") pod \"92f919b7-2340-43d5-8a2f-5d21c06586d8\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " Sep 08 16:51:26.508080 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508064 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-certs\") pod \"92f919b7-2340-43d5-8a2f-5d21c06586d8\" (UID: \"92f919b7-2340-43d5-8a2f-5d21c06586d8\") " Sep 08 16:51:26.508476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508198 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92f919b7-2340-43d5-8a2f-5d21c06586d8" (UID: "92f919b7-2340-43d5-8a2f-5d21c06586d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:51:26.508476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508309 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "92f919b7-2340-43d5-8a2f-5d21c06586d8" (UID: "92f919b7-2340-43d5-8a2f-5d21c06586d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:51:26.508476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508356 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f919b7-2340-43d5-8a2f-5d21c06586d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92f919b7-2340-43d5-8a2f-5d21c06586d8" (UID: "92f919b7-2340-43d5-8a2f-5d21c06586d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:51:26.508476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508415 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92f919b7-2340-43d5-8a2f-5d21c06586d8" (UID: "92f919b7-2340-43d5-8a2f-5d21c06586d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:51:26.508476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508473 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f919b7-2340-43d5-8a2f-5d21c06586d8-istiod-ca-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:26.508707 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508492 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-credential-socket\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:26.508707 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508497 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92f919b7-2340-43d5-8a2f-5d21c06586d8" (UID: "92f919b7-2340-43d5-8a2f-5d21c06586d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:51:26.508707 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.508508 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-data\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:26.510300 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.510278 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92f919b7-2340-43d5-8a2f-5d21c06586d8" (UID: "92f919b7-2340-43d5-8a2f-5d21c06586d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:51:26.510300 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.510290 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92f919b7-2340-43d5-8a2f-5d21c06586d8" (UID: "92f919b7-2340-43d5-8a2f-5d21c06586d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:51:26.510804 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.510783 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-kube-api-access-jhdlr" (OuterVolumeSpecName: "kube-api-access-jhdlr") pod "92f919b7-2340-43d5-8a2f-5d21c06586d8" (UID: "92f919b7-2340-43d5-8a2f-5d21c06586d8"). InnerVolumeSpecName "kube-api-access-jhdlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:51:26.510904 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.510799 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "92f919b7-2340-43d5-8a2f-5d21c06586d8" (UID: "92f919b7-2340-43d5-8a2f-5d21c06586d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:51:26.609231 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.609195 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:26.609231 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.609225 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-envoy\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:26.609231 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.609233 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-podinfo\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:26.609231 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.609240 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-istio-token\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:26.609508 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.609249 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhdlr\" (UniqueName: \"kubernetes.io/projected/92f919b7-2340-43d5-8a2f-5d21c06586d8-kube-api-access-jhdlr\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:26.609508 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:26.609258 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f919b7-2340-43d5-8a2f-5d21c06586d8-workload-socket\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:27.277115 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.277083 2567 generic.go:356] "Generic (PLEG): container finished" podID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerID="0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18" exitCode=0 Sep 08 16:51:27.277324 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.277152 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" Sep 08 16:51:27.277324 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.277166 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" event={"ID":"92f919b7-2340-43d5-8a2f-5d21c06586d8","Type":"ContainerDied","Data":"0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18"} Sep 08 16:51:27.277324 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.277219 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt" event={"ID":"92f919b7-2340-43d5-8a2f-5d21c06586d8","Type":"ContainerDied","Data":"5c785ed5c92bd8eae23b1fb265b35cfdda1f325c276056e1a85bd7166bda569a"} Sep 08 16:51:27.277324 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.277240 2567 scope.go:117] "RemoveContainer" containerID="0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18" Sep 08 16:51:27.285982 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.285961 2567 scope.go:117] "RemoveContainer" containerID="eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130" Sep 08 16:51:27.294482 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.294459 2567 scope.go:117] "RemoveContainer" containerID="0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18" Sep 08 16:51:27.294801 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:51:27.294780 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18\": container with ID starting with 0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18 not found: ID does not exist" containerID="0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18" Sep 08 16:51:27.294882 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.294809 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18"} err="failed to get container status \"0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18\": rpc error: code = NotFound desc = could not find container \"0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18\": container with ID starting with 0814a06a714cafc078dcf720218d1ff471a948063455fb3660df06519b925f18 not found: ID does not exist" Sep 08 16:51:27.294882 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.294827 2567 scope.go:117] "RemoveContainer" containerID="eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130" Sep 08 16:51:27.295119 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:51:27.295097 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130\": container with ID starting with eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130 not found: ID does not exist" containerID="eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130" Sep 08 16:51:27.295176 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.295124 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130"} err="failed to get container status \"eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130\": rpc error: code = NotFound desc = could not find container \"eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130\": container with ID starting with eb9381c7a6417f5a31e5e92b038e4b25ac15c24e14e078f7f2bd3c997bc33130 not found: ID does not exist" Sep 08 16:51:27.296366 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.296347 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt"] Sep 08 16:51:27.299858 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:27.299833 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4942b-openshift-default-65d8854976-sjgdt"] Sep 08 16:51:29.053280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:29.053249 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" path="/var/lib/kubelet/pods/92f919b7-2340-43d5-8a2f-5d21c06586d8/volumes" Sep 08 16:51:32.048070 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.047966 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj"] Sep 08 16:51:32.049856 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.049826 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-823397-29814765-9nqnj"] Sep 08 16:51:32.080795 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.080762 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764"] Sep 08 16:51:32.081077 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.081008 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" podUID="b487a056-bd11-44d7-b0a6-de200fc13fa3" containerName="payload-processing" containerID="cri-o://716bcd6b4efb84f18369c6bbe3f9e325ed4163725e9a0e4f53cd6e4006fbf6b6" gracePeriod=30 Sep 08 16:51:32.309860 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.309830 2567 generic.go:356] "Generic (PLEG): container finished" podID="b487a056-bd11-44d7-b0a6-de200fc13fa3" containerID="716bcd6b4efb84f18369c6bbe3f9e325ed4163725e9a0e4f53cd6e4006fbf6b6" exitCode=0 Sep 08 16:51:32.310061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.309873 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" event={"ID":"b487a056-bd11-44d7-b0a6-de200fc13fa3","Type":"ContainerDied","Data":"716bcd6b4efb84f18369c6bbe3f9e325ed4163725e9a0e4f53cd6e4006fbf6b6"} Sep 08 16:51:32.328324 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.328300 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:51:32.465316 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.465280 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b487a056-bd11-44d7-b0a6-de200fc13fa3-plugins-config-volume\") pod \"b487a056-bd11-44d7-b0a6-de200fc13fa3\" (UID: \"b487a056-bd11-44d7-b0a6-de200fc13fa3\") " Sep 08 16:51:32.465316 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.465321 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghd5g\" (UniqueName: \"kubernetes.io/projected/b487a056-bd11-44d7-b0a6-de200fc13fa3-kube-api-access-ghd5g\") pod \"b487a056-bd11-44d7-b0a6-de200fc13fa3\" (UID: \"b487a056-bd11-44d7-b0a6-de200fc13fa3\") " Sep 08 16:51:32.465654 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.465630 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b487a056-bd11-44d7-b0a6-de200fc13fa3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b487a056-bd11-44d7-b0a6-de200fc13fa3" (UID: "b487a056-bd11-44d7-b0a6-de200fc13fa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:51:32.467591 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.467571 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b487a056-bd11-44d7-b0a6-de200fc13fa3-kube-api-access-ghd5g" (OuterVolumeSpecName: "kube-api-access-ghd5g") pod "b487a056-bd11-44d7-b0a6-de200fc13fa3" (UID: "b487a056-bd11-44d7-b0a6-de200fc13fa3"). InnerVolumeSpecName "kube-api-access-ghd5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:51:32.566220 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.566136 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b487a056-bd11-44d7-b0a6-de200fc13fa3-plugins-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:32.566220 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:32.566165 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghd5g\" (UniqueName: \"kubernetes.io/projected/b487a056-bd11-44d7-b0a6-de200fc13fa3-kube-api-access-ghd5g\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:51:33.052457 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:33.052421 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a" path="/var/lib/kubelet/pods/d7645ad0-f3bf-4a38-afd5-5d968cdd8d9a/volumes" Sep 08 16:51:33.316576 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:33.316491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" event={"ID":"b487a056-bd11-44d7-b0a6-de200fc13fa3","Type":"ContainerDied","Data":"35a3d1f37389c8944b26ca895daf737c2a11b5b8d5ab78493726aa736312bb5f"} Sep 08 16:51:33.316576 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:33.316519 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764" Sep 08 16:51:33.316576 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:33.316551 2567 scope.go:117] "RemoveContainer" containerID="716bcd6b4efb84f18369c6bbe3f9e325ed4163725e9a0e4f53cd6e4006fbf6b6" Sep 08 16:51:33.334136 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:33.334112 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764"] Sep 08 16:51:33.342703 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:33.342678 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-823397-77b7bc5d8f-zl764"] Sep 08 16:51:35.052780 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:35.052747 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b487a056-bd11-44d7-b0a6-de200fc13fa3" path="/var/lib/kubelet/pods/b487a056-bd11-44d7-b0a6-de200fc13fa3/volumes" Sep 08 16:51:53.356764 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:53.356730 2567 scope.go:117] "RemoveContainer" containerID="6b5f38ad0a8ce907d033c21cd0313cf8818d1f4f85a90fce3d32ef053f238ead" Sep 08 16:51:53.365132 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:53.365112 2567 scope.go:117] "RemoveContainer" containerID="1e06525ff089c461f603784210a69d186266cdbafba355e45c7784a2e56b197c" Sep 08 16:51:53.372721 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:51:53.372702 2567 scope.go:117] "RemoveContainer" containerID="6bce1a071dd2bd27c1977aa499940e33ab5582bc3dd3231e1535b26f854279d6" Sep 08 16:52:53.112804 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:52:53.112774 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:52:53.119355 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:52:53.119331 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:55:53.441433 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:55:53.441399 2567 scope.go:117] "RemoveContainer" containerID="7bbb489914725856e6bbda47e304b679e136a36d6ab7cf16cecaf0c2120ee7dc" Sep 08 16:56:58.396264 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396225 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84f865c587-77hpg"] Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396598 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396608 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396617 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362d81a3-7714-4c07-ba98-7fde94dcbb1c" containerName="revoke-keys" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396622 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="362d81a3-7714-4c07-ba98-7fde94dcbb1c" containerName="revoke-keys" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396636 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396641 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396647 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b487a056-bd11-44d7-b0a6-de200fc13fa3" containerName="payload-processing" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396652 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b487a056-bd11-44d7-b0a6-de200fc13fa3" containerName="payload-processing" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396662 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da8e5a0-af62-4b02-8287-16d671f15881" containerName="payload-processing" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396667 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da8e5a0-af62-4b02-8287-16d671f15881" containerName="payload-processing" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396679 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68254c59-c198-4516-9a85-ded86eff69cf" containerName="payload-pre-processing" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396684 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="68254c59-c198-4516-9a85-ded86eff69cf" containerName="payload-pre-processing" Sep 08 16:56:58.396686 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396692 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1e6dad-4bbf-405c-a0cf-91418ea4c9af" containerName="payload-processing" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396697 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1e6dad-4bbf-405c-a0cf-91418ea4c9af" containerName="payload-processing" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396708 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba" containerName="payload-pre-processing" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396714 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba" containerName="payload-pre-processing" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396787 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1fb490-f1f3-47a2-a6d7-78ed42fa7fba" containerName="payload-pre-processing" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396794 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396802 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da8e5a0-af62-4b02-8287-16d671f15881" containerName="payload-processing" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396807 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b487a056-bd11-44d7-b0a6-de200fc13fa3" containerName="payload-processing" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396813 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="362d81a3-7714-4c07-ba98-7fde94dcbb1c" containerName="revoke-keys" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396818 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="68254c59-c198-4516-9a85-ded86eff69cf" containerName="payload-pre-processing" Sep 08 16:56:58.397163 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.396824 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1e6dad-4bbf-405c-a0cf-91418ea4c9af" containerName="payload-processing" Sep 08 16:56:58.399902 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.399886 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.403847 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.403818 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 16:56:58.403971 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.403845 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 16:56:58.403971 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.403844 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 16:56:58.404256 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.404237 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-65b2g\"" Sep 08 16:56:58.404871 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.404856 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:56:58.418451 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.418430 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f865c587-77hpg"] Sep 08 16:56:58.521739 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.521705 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-maas-api-tls\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.521946 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.521758 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514e7b1f-c54c-4bc5-81f7-8a2459df2148-openshift-service-ca\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.521946 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.521877 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2sdn\" (UniqueName: \"kubernetes.io/projected/514e7b1f-c54c-4bc5-81f7-8a2459df2148-kube-api-access-d2sdn\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.522098 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.521997 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-metrics-certs\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.622963 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.622915 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-metrics-certs\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.622963 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.622967 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-maas-api-tls\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.623222 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.623006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514e7b1f-c54c-4bc5-81f7-8a2459df2148-openshift-service-ca\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.623222 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.623089 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2sdn\" (UniqueName: \"kubernetes.io/projected/514e7b1f-c54c-4bc5-81f7-8a2459df2148-kube-api-access-d2sdn\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.623800 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.623723 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514e7b1f-c54c-4bc5-81f7-8a2459df2148-openshift-service-ca\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.625609 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.625582 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-metrics-certs\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.625708 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.625662 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-maas-api-tls\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.631809 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.631782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2sdn\" (UniqueName: \"kubernetes.io/projected/514e7b1f-c54c-4bc5-81f7-8a2459df2148-kube-api-access-d2sdn\") pod \"maas-api-84f865c587-77hpg\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.710844 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.710813 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:56:58.996386 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:58.996342 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f865c587-77hpg"] Sep 08 16:56:58.998882 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:56:58.998854 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514e7b1f_c54c_4bc5_81f7_8a2459df2148.slice/crio-165f4e0cbcace9efeea8fc43e525c079ffa9d37ede856128c5361140957b3fa4 WatchSource:0}: Error finding container 165f4e0cbcace9efeea8fc43e525c079ffa9d37ede856128c5361140957b3fa4: Status 404 returned error can't find the container with id 165f4e0cbcace9efeea8fc43e525c079ffa9d37ede856128c5361140957b3fa4 Sep 08 16:56:59.007945 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:59.007913 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:56:59.921627 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:56:59.921586 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" event={"ID":"514e7b1f-c54c-4bc5-81f7-8a2459df2148","Type":"ContainerStarted","Data":"165f4e0cbcace9efeea8fc43e525c079ffa9d37ede856128c5361140957b3fa4"} Sep 08 16:57:00.929033 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:00.928982 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" event={"ID":"514e7b1f-c54c-4bc5-81f7-8a2459df2148","Type":"ContainerStarted","Data":"4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa"} Sep 08 16:57:00.929491 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:00.929078 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:57:00.952943 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:00.952893 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" podStartSLOduration=1.176233367 podStartE2EDuration="2.95287928s" totalImagesPullingTime="1.776645913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:00.952284518 +0000 UTC m=+2048.423270975" watchObservedRunningTime="2026-09-08 16:57:00.95287928 +0000 UTC m=+2048.423865736" Sep 08 16:57:06.051494 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.051463 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f865c587-77hpg"] Sep 08 16:57:06.051927 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.051630 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" podUID="514e7b1f-c54c-4bc5-81f7-8a2459df2148" containerName="maas-api" containerID="cri-o://4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa" gracePeriod=30 Sep 08 16:57:06.056681 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.056638 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:57:06.294030 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.293994 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:57:06.397888 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.397799 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2sdn\" (UniqueName: \"kubernetes.io/projected/514e7b1f-c54c-4bc5-81f7-8a2459df2148-kube-api-access-d2sdn\") pod \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " Sep 08 16:57:06.397888 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.397870 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-metrics-certs\") pod \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " Sep 08 16:57:06.398130 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.397924 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514e7b1f-c54c-4bc5-81f7-8a2459df2148-openshift-service-ca\") pod \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " Sep 08 16:57:06.398130 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.398006 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-maas-api-tls\") pod \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\" (UID: \"514e7b1f-c54c-4bc5-81f7-8a2459df2148\") " Sep 08 16:57:06.398352 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.398329 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514e7b1f-c54c-4bc5-81f7-8a2459df2148-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "514e7b1f-c54c-4bc5-81f7-8a2459df2148" (UID: "514e7b1f-c54c-4bc5-81f7-8a2459df2148"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:57:06.400224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.400188 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "514e7b1f-c54c-4bc5-81f7-8a2459df2148" (UID: "514e7b1f-c54c-4bc5-81f7-8a2459df2148"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:57:06.400224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.400190 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514e7b1f-c54c-4bc5-81f7-8a2459df2148-kube-api-access-d2sdn" (OuterVolumeSpecName: "kube-api-access-d2sdn") pod "514e7b1f-c54c-4bc5-81f7-8a2459df2148" (UID: "514e7b1f-c54c-4bc5-81f7-8a2459df2148"). InnerVolumeSpecName "kube-api-access-d2sdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:57:06.400224 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.400208 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "514e7b1f-c54c-4bc5-81f7-8a2459df2148" (UID: "514e7b1f-c54c-4bc5-81f7-8a2459df2148"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:57:06.499149 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.499117 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-metrics-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:57:06.499149 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.499144 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514e7b1f-c54c-4bc5-81f7-8a2459df2148-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:57:06.499149 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.499153 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514e7b1f-c54c-4bc5-81f7-8a2459df2148-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:57:06.499383 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.499163 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2sdn\" (UniqueName: \"kubernetes.io/projected/514e7b1f-c54c-4bc5-81f7-8a2459df2148-kube-api-access-d2sdn\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\"" Sep 08 16:57:06.959481 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.959442 2567 generic.go:356] "Generic (PLEG): container finished" podID="514e7b1f-c54c-4bc5-81f7-8a2459df2148" containerID="4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa" exitCode=0 Sep 08 16:57:06.959688 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.959489 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" Sep 08 16:57:06.959688 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.959512 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" event={"ID":"514e7b1f-c54c-4bc5-81f7-8a2459df2148","Type":"ContainerDied","Data":"4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa"} Sep 08 16:57:06.959688 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.959564 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f865c587-77hpg" event={"ID":"514e7b1f-c54c-4bc5-81f7-8a2459df2148","Type":"ContainerDied","Data":"165f4e0cbcace9efeea8fc43e525c079ffa9d37ede856128c5361140957b3fa4"} Sep 08 16:57:06.959688 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.959591 2567 scope.go:117] "RemoveContainer" containerID="4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa" Sep 08 16:57:06.968947 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.968928 2567 scope.go:117] "RemoveContainer" containerID="4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa" Sep 08 16:57:06.969235 ip-10-0-128-131 kubenswrapper[2567]: E0908 16:57:06.969216 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa\": container with ID starting with 4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa not found: ID does not exist" containerID="4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa" Sep 08 16:57:06.969285 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.969245 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa"} err="failed to get container status \"4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa\": rpc error: code = NotFound desc = could not find container \"4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa\": container with ID starting with 4507c95e850689d3665654593f4de039eb404f708778b2479a32593f1b236dfa not found: ID does not exist" Sep 08 16:57:06.983041 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.982984 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f865c587-77hpg"] Sep 08 16:57:06.988330 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:06.988310 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f865c587-77hpg"] Sep 08 16:57:07.053229 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:07.053200 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="514e7b1f-c54c-4bc5-81f7-8a2459df2148" path="/var/lib/kubelet/pods/514e7b1f-c54c-4bc5-81f7-8a2459df2148/volumes" Sep 08 16:57:28.699267 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:28.699184 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cbd679f45-jbrwq_2be41c57-7cf8-4471-a627-a801a33f7588/authorino/0.log" Sep 08 16:57:34.852610 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:34.852576 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fcf9b86-lwglg_6bf0430c-4800-48e9-925b-55bf8fae9ff8/copy-manifests/0.log" Sep 08 16:57:34.860263 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:34.860237 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fcf9b86-lwglg_6bf0430c-4800-48e9-925b-55bf8fae9ff8/manager/0.log" Sep 08 16:57:36.327155 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:36.327127 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cbd679f45-jbrwq_2be41c57-7cf8-4471-a627-a801a33f7588/authorino/0.log" Sep 08 16:57:36.444154 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:36.444122 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-4smmm_38899c90-da56-42ba-9d43-1958658bd59e/manager/0.log" Sep 08 16:57:37.659779 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:37.659751 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56cf6b7cfc-nlzbq_f03454b1-e2b6-4fcc-927d-b47cb1f0a899/kube-auth-proxy/0.log" Sep 08 16:57:38.031386 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:38.031302 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ltrmf_1f0a8495-338d-4e24-a0af-e4fc2f9f0be1/payload-pre-processing/0.log" Sep 08 16:57:38.843746 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:38.843720 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-f96rm_f8734e94-31dc-41fb-ba8c-b40fe657a174/main/0.log" Sep 08 16:57:39.068030 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:39.067992 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz4jxb_d710d22b-c6e8-4658-8249-81c800e5a2e0/main/0.log" Sep 08 16:57:39.297061 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:39.297034 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cfghv_d7ab6632-24a2-4d65-b78e-3e63f23216c0/main/0.log" Sep 08 16:57:48.385625 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:48.385592 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cqvbk_a3e8e45e-b772-4cc4-87c8-f9222bdd5488/global-pull-secret-syncer/0.log" Sep 08 16:57:48.604971 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:48.604943 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-td9m4_f9ac825f-6d94-455d-a06b-2f3674222f6c/konnectivity-agent/0.log" Sep 08 16:57:48.631269 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:48.631238 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-131.ec2.internal_8f25ceed021acd67419e766168870646/haproxy/0.log" Sep 08 16:57:53.148035 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:53.147991 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:57:53.155490 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:53.155467 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vfmw_17ca8c4a-6567-4610-9166-f37f64cb40a5/ovn-acl-logging/0.log" Sep 08 16:57:53.615585 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:53.615543 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cbd679f45-jbrwq_2be41c57-7cf8-4471-a627-a801a33f7588/authorino/0.log" Sep 08 16:57:53.661240 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:53.661194 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-4smmm_38899c90-da56-42ba-9d43-1958658bd59e/manager/0.log" Sep 08 16:57:55.491194 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.491166 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37ef7003-0710-4399-94e9-db4fe1adcffa/alertmanager/0.log" Sep 08 16:57:55.521432 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.521398 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37ef7003-0710-4399-94e9-db4fe1adcffa/config-reloader/0.log" Sep 08 16:57:55.559355 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.559336 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37ef7003-0710-4399-94e9-db4fe1adcffa/kube-rbac-proxy-web/0.log" Sep 08 16:57:55.592152 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.592126 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37ef7003-0710-4399-94e9-db4fe1adcffa/kube-rbac-proxy/0.log" Sep 08 16:57:55.614854 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.614796 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37ef7003-0710-4399-94e9-db4fe1adcffa/kube-rbac-proxy-metric/0.log" Sep 08 16:57:55.642495 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.642474 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37ef7003-0710-4399-94e9-db4fe1adcffa/prom-label-proxy/0.log" Sep 08 16:57:55.670859 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.670839 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37ef7003-0710-4399-94e9-db4fe1adcffa/init-config-reloader/0.log" Sep 08 16:57:55.751110 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.751081 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gc886_c92f55be-0da5-4c92-81bf-93ae92a68386/kube-state-metrics/0.log" Sep 08 16:57:55.772656 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.772631 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gc886_c92f55be-0da5-4c92-81bf-93ae92a68386/kube-rbac-proxy-main/0.log" Sep 08 16:57:55.796569 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.796545 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gc886_c92f55be-0da5-4c92-81bf-93ae92a68386/kube-rbac-proxy-self/0.log" Sep 08 16:57:55.834665 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.834644 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fc5949489-w2wk8_a335b4f2-2d52-4f21-95ad-c4f76259c0db/metrics-server/0.log" Sep 08 16:57:55.873310 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.873246 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-sqqpm_614b8d36-dbe9-49fb-b0c0-5da6e73987ea/monitoring-plugin/0.log" Sep 08 16:57:55.912678 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.912658 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-58zzz_97526267-ea08-4c82-aa89-5aeee4646510/node-exporter/0.log" Sep 08 16:57:55.945280 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.945256 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-58zzz_97526267-ea08-4c82-aa89-5aeee4646510/kube-rbac-proxy/0.log" Sep 08 16:57:55.974264 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:55.974243 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-58zzz_97526267-ea08-4c82-aa89-5aeee4646510/init-textfile/0.log" Sep 08 16:57:56.208901 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.208877 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xs9n4_bfb80ebb-1763-4fa4-9d57-3bb46df66d79/kube-rbac-proxy-main/0.log" Sep 08 16:57:56.234057 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.234035 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xs9n4_bfb80ebb-1763-4fa4-9d57-3bb46df66d79/kube-rbac-proxy-self/0.log" Sep 08 16:57:56.260999 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.260978 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xs9n4_bfb80ebb-1763-4fa4-9d57-3bb46df66d79/openshift-state-metrics/0.log" Sep 08 16:57:56.313226 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.313203 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_117aaedb-2d68-4522-8526-e4c19471f9d1/prometheus/0.log" Sep 08 16:57:56.345566 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.345531 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_117aaedb-2d68-4522-8526-e4c19471f9d1/config-reloader/0.log" Sep 08 16:57:56.413532 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.413506 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_117aaedb-2d68-4522-8526-e4c19471f9d1/thanos-sidecar/0.log" Sep 08 16:57:56.447701 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.447665 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_117aaedb-2d68-4522-8526-e4c19471f9d1/kube-rbac-proxy-web/0.log" Sep 08 16:57:56.478622 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.478604 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_117aaedb-2d68-4522-8526-e4c19471f9d1/kube-rbac-proxy/0.log" Sep 08 16:57:56.516300 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.516275 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_117aaedb-2d68-4522-8526-e4c19471f9d1/kube-rbac-proxy-thanos/0.log" Sep 08 16:57:56.550461 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.550442 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_117aaedb-2d68-4522-8526-e4c19471f9d1/init-config-reloader/0.log" Sep 08 16:57:56.600348 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.600319 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wbhl9_f4fa655f-11ee-49c0-9f94-914abaf95664/prometheus-operator/0.log" Sep 08 16:57:56.625972 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.625946 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wbhl9_f4fa655f-11ee-49c0-9f94-914abaf95664/kube-rbac-proxy/0.log" Sep 08 16:57:56.714787 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.714707 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d74d56774-hkgxr_913426d9-bf7f-443e-8da2-d68e6b953e1f/telemeter-client/0.log" Sep 08 16:57:56.769476 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.769450 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d74d56774-hkgxr_913426d9-bf7f-443e-8da2-d68e6b953e1f/reload/0.log" Sep 08 16:57:56.833524 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.833501 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d74d56774-hkgxr_913426d9-bf7f-443e-8da2-d68e6b953e1f/kube-rbac-proxy/0.log" Sep 08 16:57:56.899346 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.899321 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/thanos-query/0.log" Sep 08 16:57:56.962376 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:56.962354 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/kube-rbac-proxy-web/0.log" Sep 08 16:57:57.033161 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:57.033089 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/kube-rbac-proxy/0.log" Sep 08 16:57:57.080264 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:57.080239 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/prom-label-proxy/0.log" Sep 08 16:57:57.125573 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:57.125540 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/kube-rbac-proxy-rules/0.log" Sep 08 16:57:57.169165 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:57.169141 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d88d5644f-x68ct_f517b143-d4bb-496d-9838-c554d4babd5b/kube-rbac-proxy-metrics/0.log" Sep 08 16:57:58.687073 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.687010 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv"] Sep 08 16:57:58.687483 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.687469 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514e7b1f-c54c-4bc5-81f7-8a2459df2148" containerName="maas-api" Sep 08 16:57:58.687525 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.687485 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="514e7b1f-c54c-4bc5-81f7-8a2459df2148" containerName="maas-api" Sep 08 16:57:58.687561 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.687545 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="514e7b1f-c54c-4bc5-81f7-8a2459df2148" containerName="maas-api" Sep 08 16:57:58.687561 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.687553 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f919b7-2340-43d5-8a2f-5d21c06586d8" containerName="istio-proxy" Sep 08 16:57:58.690683 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.690665 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.694500 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.694469 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t96w8\"/\"kube-root-ca.crt\"" Sep 08 16:57:58.694643 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.694615 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t96w8\"/\"default-dockercfg-52jnx\"" Sep 08 16:57:58.695351 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.695338 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t96w8\"/\"openshift-service-ca.crt\"" Sep 08 16:57:58.769193 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.769160 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv"] Sep 08 16:57:58.780339 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.780314 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftjp\" (UniqueName: \"kubernetes.io/projected/43d352a9-b8d7-48b9-b334-4d6d4602e82f-kube-api-access-xftjp\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.780490 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.780362 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-podres\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.780490 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.780428 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-lib-modules\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.780571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.780539 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-sys\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.780571 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.780562 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-proc\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.881609 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.881575 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-sys\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.881609 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.881611 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-proc\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.881826 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.881642 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftjp\" (UniqueName: \"kubernetes.io/projected/43d352a9-b8d7-48b9-b334-4d6d4602e82f-kube-api-access-xftjp\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.881826 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.881686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-podres\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.881826 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.881707 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-lib-modules\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.881826 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.881720 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-sys\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.881826 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.881720 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-proc\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.881988 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.881838 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-podres\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.881988 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.881844 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43d352a9-b8d7-48b9-b334-4d6d4602e82f-lib-modules\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:58.893840 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:58.893806 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftjp\" (UniqueName: \"kubernetes.io/projected/43d352a9-b8d7-48b9-b334-4d6d4602e82f-kube-api-access-xftjp\") pod \"perf-node-gather-daemonset-fqzfv\" (UID: \"43d352a9-b8d7-48b9-b334-4d6d4602e82f\") " pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:59.000373 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:59.000284 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:57:59.295527 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:59.295444 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv"] Sep 08 16:57:59.299216 ip-10-0-128-131 kubenswrapper[2567]: W0908 16:57:59.299190 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod43d352a9_b8d7_48b9_b334_4d6d4602e82f.slice/crio-d3f592182c8e727a53d659342edfda95c8f7cff0082010fa40dd102ae62eebed WatchSource:0}: Error finding container d3f592182c8e727a53d659342edfda95c8f7cff0082010fa40dd102ae62eebed: Status 404 returned error can't find the container with id d3f592182c8e727a53d659342edfda95c8f7cff0082010fa40dd102ae62eebed Sep 08 16:57:59.988863 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:57:59.988828 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-thmg6_2e685fc5-c9a0-4df7-bdcb-1ef50eaf3eed/download-server/0.log" Sep 08 16:58:00.221292 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:00.221258 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" event={"ID":"43d352a9-b8d7-48b9-b334-4d6d4602e82f","Type":"ContainerStarted","Data":"821b1ec21ebf502d927b31ae7b79014f61aea6726f1e5443cdfed0b5769a0b00"} Sep 08 16:58:00.221484 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:00.221311 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" event={"ID":"43d352a9-b8d7-48b9-b334-4d6d4602e82f","Type":"ContainerStarted","Data":"d3f592182c8e727a53d659342edfda95c8f7cff0082010fa40dd102ae62eebed"} Sep 08 16:58:00.221484 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:00.221348 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:58:00.249628 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:00.249539 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" podStartSLOduration=2.249524173 podStartE2EDuration="2.249524173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:58:00.246965865 +0000 UTC m=+2107.717952322" watchObservedRunningTime="2026-09-08 16:58:00.249524173 +0000 UTC m=+2107.720510628" Sep 08 16:58:02.164357 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:02.164325 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q8bwr_f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26/dns/0.log" Sep 08 16:58:02.200367 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:02.200335 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q8bwr_f4d56ec0-c9ed-44b1-99b3-bfd33db6ee26/kube-rbac-proxy/0.log" Sep 08 16:58:02.273884 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:02.273856 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dkg7m_40daeca6-4624-45bf-831f-51e77aab369c/dns-node-resolver/0.log" Sep 08 16:58:03.022338 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:03.022300 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5grg2_3ec20b4a-239d-49aa-b713-e65f0ba60775/node-ca/0.log" Sep 08 16:58:04.386629 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:04.386600 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56cf6b7cfc-nlzbq_f03454b1-e2b6-4fcc-927d-b47cb1f0a899/kube-auth-proxy/0.log" Sep 08 16:58:04.508290 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:04.508257 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ltrmf_1f0a8495-338d-4e24-a0af-e4fc2f9f0be1/payload-pre-processing/0.log" Sep 08 16:58:05.599640 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:05.599611 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zvhbh_99976398-38d1-4613-891d-cb52ffd9af35/serve-healthcheck-canary/0.log" Sep 08 16:58:06.237464 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:06.237433 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t96w8/perf-node-gather-daemonset-fqzfv" Sep 08 16:58:06.289871 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:06.289844 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dm64v_a892e068-3486-42a1-8dff-472b490d9721/kube-rbac-proxy/0.log" Sep 08 16:58:06.340812 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:06.340788 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dm64v_a892e068-3486-42a1-8dff-472b490d9721/exporter/0.log" Sep 08 16:58:06.372999 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:06.372969 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dm64v_a892e068-3486-42a1-8dff-472b490d9721/extractor/0.log" Sep 08 16:58:08.994806 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:08.994771 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56cf6b7cfc-nlzbq_f03454b1-e2b6-4fcc-927d-b47cb1f0a899/kube-auth-proxy/0.log" Sep 08 16:58:09.066504 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:09.066472 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ltrmf_1f0a8495-338d-4e24-a0af-e4fc2f9f0be1/payload-pre-processing/0.log" Sep 08 16:58:09.805252 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:09.805225 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fcf9b86-lwglg_6bf0430c-4800-48e9-925b-55bf8fae9ff8/manager/0.log" Sep 08 16:58:09.836568 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:09.836545 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fcf9b86-lwglg_6bf0430c-4800-48e9-925b-55bf8fae9ff8/copy-manifests/0.log" Sep 08 16:58:10.811125 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:10.811090 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56cf6b7cfc-nlzbq_f03454b1-e2b6-4fcc-927d-b47cb1f0a899/kube-auth-proxy/0.log" Sep 08 16:58:10.895256 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:10.895217 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ltrmf_1f0a8495-338d-4e24-a0af-e4fc2f9f0be1/payload-pre-processing/0.log" Sep 08 16:58:11.112588 ip-10-0-128-131 kubenswrapper[2567]: I0908 16:58:11.112523 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5ddb9f5cb8-lkz9j_8a13e4b9-6153-46fc-adb3-ff09dbe153d1/manager/0.log"