Jul 28 22:04:35.237238 ip-10-0-142-225 systemd[1]: Starting Kubernetes Kubelet... Jul 28 22:04:35.675448 ip-10-0-142-225 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 22:04:35.675448 ip-10-0-142-225 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 22:04:35.675448 ip-10-0-142-225 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 22:04:35.675448 ip-10-0-142-225 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 22:04:35.675448 ip-10-0-142-225 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 22:04:35.676493 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.676391 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 22:04:35.681535 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681509 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:04:35.681535 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681530 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:04:35.681535 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681536 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:04:35.681535 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681540 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:04:35.681535 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681545 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681549 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681554 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681559 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681564 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681568 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681574 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681578 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681581 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681585 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681590 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681594 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681599 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681603 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681608 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681611 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681615 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681619 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681623 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681627 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:04:35.681825 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681633 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681637 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681641 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681645 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681650 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681654 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681658 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681662 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681667 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681671 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681675 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681679 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681684 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681688 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681694 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681699 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681704 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681710 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681715 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681719 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:04:35.682657 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681724 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681728 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681732 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681737 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681776 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681794 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681798 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681801 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681806 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681812 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681815 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681819 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681822 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681827 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681832 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681836 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681839 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681843 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681846 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:04:35.683560 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681849 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681852 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681855 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681858 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681861 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681864 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681867 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681870 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681873 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681876 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681879 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681882 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681885 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681887 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681891 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681894 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681897 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681900 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.681966 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682030 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:04:35.684185 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682034 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682045 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682049 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682053 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682061 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682068 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682075 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682082 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682087 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.682093 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.683382 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.683423 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.683444 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.683449 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.683583 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.683590 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684448 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684457 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:04:35.684707 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684460 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684463 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684466 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684469 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684473 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684476 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684479 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684482 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684485 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684487 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684490 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684493 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684495 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684498 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684501 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684504 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684507 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684510 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684513 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:04:35.685202 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684518 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684524 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684527 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684532 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684535 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684538 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684542 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684545 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684547 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684550 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684553 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684556 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684558 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684561 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684564 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684568 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684570 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684574 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684576 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684579 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:04:35.685737 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684582 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684584 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684587 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684591 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684593 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684597 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684600 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684603 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684607 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684612 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684614 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684618 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684621 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684624 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684627 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684630 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684633 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684636 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684639 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:04:35.686305 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684642 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684644 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684647 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684650 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684653 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684656 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684659 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684662 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684664 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684669 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684672 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684674 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684677 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684680 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684683 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684685 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684688 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684691 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684693 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684696 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:04:35.686815 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684699 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684703 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684706 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684708 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684711 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684714 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684717 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684720 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684723 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684726 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684729 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684733 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684736 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684739 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684743 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684747 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684750 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684752 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.684755 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685270 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:04:35.687365 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685278 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685281 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685285 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685288 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685291 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685294 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685297 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685300 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685304 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685307 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685310 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685313 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685317 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685320 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685323 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685327 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685330 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685333 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685336 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685338 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:04:35.687892 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685341 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685344 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685348 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685352 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685355 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685358 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685361 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685363 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685366 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685369 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685372 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685375 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685378 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685381 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685383 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685386 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685389 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685392 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685394 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685397 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:04:35.688457 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685399 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685402 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685405 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685408 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685410 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685413 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685416 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685420 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685422 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685425 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685428 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685431 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685433 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685436 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685439 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685441 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685444 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685446 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685449 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:04:35.688988 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685452 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685455 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685458 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685463 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685467 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685471 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685474 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685477 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685479 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685482 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685485 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685488 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685490 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685494 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685497 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685499 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685502 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685505 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685508 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:04:35.689511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685511 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685514 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685517 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685520 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685523 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685526 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685530 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685533 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685537 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685540 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685542 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685545 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685548 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685550 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685553 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685556 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685558 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685561 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685564 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.685567 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:04:35.690064 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685642 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685651 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685658 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685663 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685668 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685671 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685676 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685681 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685684 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685688 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685692 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685696 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685699 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685704 2557 flags.go:64] FLAG: --cgroup-root="" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685707 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685710 2557 flags.go:64] FLAG: --client-ca-file="" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685714 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685717 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685722 2557 flags.go:64] FLAG: --cluster-domain="" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685725 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685728 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685732 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685736 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685740 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 22:04:35.690607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685744 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685747 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685751 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685755 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685758 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685762 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685765 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685769 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685774 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685777 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685780 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685783 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685786 2557 flags.go:64] FLAG: --enable-server="true" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685789 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685794 2557 flags.go:64] FLAG: --event-burst="100" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685798 2557 flags.go:64] FLAG: --event-qps="50" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685801 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685804 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685807 2557 flags.go:64] FLAG: --eviction-hard="" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685812 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685815 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685819 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685822 2557 flags.go:64] FLAG: --eviction-soft="" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685825 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685839 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 22:04:35.691249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685842 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685846 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685849 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685852 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685855 2557 flags.go:64] FLAG: --feature-gates="" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685862 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685865 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685869 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685872 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685875 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685878 2557 flags.go:64] FLAG: --help="false" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685882 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-142-225.ec2.internal" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685885 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685888 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685891 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685896 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685900 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685903 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685906 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685909 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685916 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685919 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685923 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685926 2557 flags.go:64] FLAG: --kube-reserved="" Jul 28 22:04:35.691971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685929 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685932 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685935 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685938 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685942 2557 flags.go:64] FLAG: --lock-file="" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685944 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685948 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685951 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685957 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685960 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685963 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685967 2557 flags.go:64] FLAG: --logging-format="text" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685970 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685974 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685977 2557 flags.go:64] FLAG: --manifest-url="" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685980 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685985 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685988 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685992 2557 flags.go:64] FLAG: --max-pods="110" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.685996 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686014 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686018 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686021 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686025 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686028 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 22:04:35.692635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686031 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686040 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686043 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686048 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686051 2557 flags.go:64] FLAG: --pod-cidr="" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686054 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686060 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686063 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686066 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686069 2557 flags.go:64] FLAG: --port="10250" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686073 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686076 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-019d6f5f6a4e8f7ef" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686080 2557 flags.go:64] FLAG: --qos-reserved="" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686083 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686086 2557 flags.go:64] FLAG: --register-node="true" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686089 2557 flags.go:64] FLAG: --register-with-taints="" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686093 2557 flags.go:64] FLAG: --registry-burst="10" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686096 2557 flags.go:64] FLAG: --registry-qps="5" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686100 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686102 2557 flags.go:64] FLAG: --reserved-memory="" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686106 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686110 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686113 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686116 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686119 2557 flags.go:64] FLAG: --runonce="false" Jul 28 22:04:35.693317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686122 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686126 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686129 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686132 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686135 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686138 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686141 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686145 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686148 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686152 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686156 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686159 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686162 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686166 2557 flags.go:64] FLAG: --system-cgroups="" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686169 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686175 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686178 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686181 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686186 2557 flags.go:64] FLAG: --tls-min-version="" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686189 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686192 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686195 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686199 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686202 2557 flags.go:64] FLAG: --v="2" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686207 2557 flags.go:64] FLAG: --version="false" Jul 28 22:04:35.694016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686211 2557 flags.go:64] FLAG: --vmodule="" Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686215 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.686219 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686349 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686354 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686358 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686361 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686364 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686367 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686370 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686381 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686384 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686387 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686390 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686393 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686397 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686400 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686404 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686407 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686410 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:04:35.694690 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686416 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686418 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686421 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686424 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686427 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686430 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686435 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686438 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686442 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686445 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686448 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686451 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686454 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686457 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686460 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686463 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686466 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686469 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686472 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:04:35.695247 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686475 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686478 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686480 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686483 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686486 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686489 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686492 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686495 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686497 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686501 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686504 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686506 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686509 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686513 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686516 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686519 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686522 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686525 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686527 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686530 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:04:35.695831 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686533 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686535 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686538 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686541 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686544 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686546 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686550 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686553 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686555 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686558 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686561 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686564 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686566 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686569 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686572 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686574 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686577 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686580 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686582 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:04:35.696473 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686585 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686588 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686591 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686594 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686597 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686600 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686604 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686606 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686609 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686612 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686615 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686619 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686623 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686625 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686630 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686633 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686636 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686639 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686642 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686645 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:04:35.696976 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686649 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686652 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686654 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.686658 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.687699 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.695158 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.695176 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.695197 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.695206 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695270 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695276 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695279 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695283 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695286 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695289 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:04:35.697511 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695293 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695296 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695299 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695302 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695305 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695308 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695312 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695315 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695319 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695322 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695325 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695328 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695332 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695335 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695337 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695340 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695343 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695346 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695349 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695352 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:04:35.697914 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695355 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695358 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695361 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695366 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695370 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695373 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695376 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695380 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695383 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695386 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695389 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695391 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695394 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695398 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695400 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695403 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695406 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695409 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695412 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:04:35.698466 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695414 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695417 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695420 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695422 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695425 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695428 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695430 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695435 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695438 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695441 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695444 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695448 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695451 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695453 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695457 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695460 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695463 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695467 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695470 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:04:35.698964 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695472 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695475 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695478 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695481 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695484 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695486 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695489 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695492 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695495 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695498 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695500 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695503 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695506 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695509 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695511 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695514 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695517 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695520 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695522 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695525 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:04:35.699503 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695528 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695530 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695533 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695536 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695538 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695541 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695544 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695547 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695550 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695553 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695556 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695558 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695563 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695566 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695569 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:04:35.700048 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.695575 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695697 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695703 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695707 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695710 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695714 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695718 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695721 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695724 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695727 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695730 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695734 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695737 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695740 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695742 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695745 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695748 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695751 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695754 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:04:35.700450 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695757 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695760 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695763 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695765 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695768 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695771 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695775 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695778 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695780 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695783 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695785 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695788 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695791 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695793 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695796 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695799 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695801 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695804 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695807 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695809 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:04:35.700954 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695812 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695815 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695818 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695820 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695823 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695826 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695829 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695831 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695834 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695837 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695840 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695843 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695846 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695849 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695851 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695854 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695856 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695859 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695862 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695865 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:04:35.701497 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695868 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695870 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695873 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695876 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695879 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695881 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695884 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695887 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695889 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695893 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695896 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695899 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695902 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695905 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695908 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695911 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695913 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695916 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695919 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695921 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:04:35.702031 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695924 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695927 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695930 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695933 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695935 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695938 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695940 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695943 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695945 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695948 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695952 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695956 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695959 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695963 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695966 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695969 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695972 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695975 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695978 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:04:35.702588 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695981 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:04:35.703101 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:35.695984 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:04:35.703101 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.695989 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 22:04:35.703101 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.696129 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 22:04:35.703101 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.698402 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 22:04:35.703101 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.699727 2557 server.go:1019] "Starting client certificate rotation" Jul 28 22:04:35.703101 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.699840 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 22:04:35.703101 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.699876 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 22:04:35.703101 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.702303 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 22:04:35.705336 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.705318 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 22:04:35.719855 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.719829 2557 log.go:25] "Validated CRI v1 runtime API" Jul 28 22:04:35.722119 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.722104 2557 log.go:25] "Validated CRI v1 image API" Jul 28 22:04:35.722957 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.722928 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 22:04:35.727179 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.727159 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 22:04:35.728209 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.728187 2557 fs.go:135] Filesystem UUIDs: map[4c21ccef-12f8-4a96-8027-3a01560c2e6e:/dev/nvme0n1p3 73ba61fb-fb4b-481a-b908-c8bc4f099010:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 28 22:04:35.728279 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.728208 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 22:04:35.734710 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.734590 2557 manager.go:216] Machine: {Timestamp:2026-07-28 22:04:35.733143722 +0000 UTC m=+0.371002844 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3082144 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ba1eefe46d4c2f28444f9f21085ae SystemUUID:ec2ba1ee-fe46-d4c2-f284-44f9f21085ae BootID:248a6040-58d7-4221-84ff-97c884c95ad5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e1:b5:e8:29:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e1:b5:e8:29:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:04:21:8e:f9:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 22:04:35.734710 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.734707 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 22:04:35.734852 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.734839 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 22:04:35.738461 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.738435 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 22:04:35.738622 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.738464 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-225.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 22:04:35.740088 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.740077 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 22:04:35.740128 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.740090 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 22:04:35.740128 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.740099 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 22:04:35.740884 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.740873 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 22:04:35.741765 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.741755 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 22:04:35.743839 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.743827 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 28 22:04:35.743954 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.743944 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 22:04:35.743996 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.743977 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 28 22:04:35.743996 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.743988 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 22:04:35.744089 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.744017 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 22:04:35.744089 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.744028 2557 kubelet.go:398] "Adding apiserver pod source" Jul 28 22:04:35.744089 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.744038 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 22:04:35.744437 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.744424 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 22:04:35.744490 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.744445 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 22:04:35.746960 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.746941 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 22:04:35.749018 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.748989 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 22:04:35.749081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.749035 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 22:04:35.750767 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750754 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 22:04:35.750816 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750774 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 22:04:35.750816 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750782 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 22:04:35.750816 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750790 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 22:04:35.750816 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750805 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 22:04:35.750816 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750814 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 22:04:35.750970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750827 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 22:04:35.750970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750836 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 22:04:35.750970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750847 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 22:04:35.750970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750859 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 22:04:35.750970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750871 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 22:04:35.750970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.750876 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 22:04:35.752378 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.752363 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 22:04:35.752378 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.752374 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 22:04:35.753543 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.753516 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 22:04:35.754464 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.754445 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d8n6k" Jul 28 22:04:35.755996 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.755941 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 22:04:35.758343 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.758328 2557 server.go:1268] "Started kubelet" Jul 28 22:04:35.758651 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.758549 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 22:04:35.758651 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.758525 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 22:04:35.758780 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.758766 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-225.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 22:04:35.758834 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.758798 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 22:04:35.759375 ip-10-0-142-225 systemd[1]: Started Kubernetes Kubelet. Jul 28 22:04:35.763950 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.763917 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 22:04:35.765393 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.765371 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d8n6k" Jul 28 22:04:35.766120 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.765619 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 22:04:35.767461 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.767443 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 28 22:04:35.770222 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.767075 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-225.ec2.internal.18c6935062815fcb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-225.ec2.internal,UID:ip-10-0-142-225.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-225.ec2.internal,},FirstTimestamp:2026-07-28 22:04:35.758292939 +0000 UTC m=+0.396152062,LastTimestamp:2026-07-28 22:04:35.758292939 +0000 UTC m=+0.396152062,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-225.ec2.internal,}" Jul 28 22:04:35.775109 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.775086 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 22:04:35.775109 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.775096 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 22:04:35.775492 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.775463 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:35.775960 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.775923 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 22:04:35.776093 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776076 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 22:04:35.776206 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.775963 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 22:04:35.776259 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776204 2557 factory.go:55] Registering systemd factory Jul 28 22:04:35.776259 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776221 2557 factory.go:223] Registration of the systemd container factory successfully Jul 28 22:04:35.776259 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776220 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 28 22:04:35.776383 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776265 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 28 22:04:35.776466 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776452 2557 factory.go:153] Registering CRI-O factory Jul 28 22:04:35.776466 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776467 2557 factory.go:223] Registration of the crio container factory successfully Jul 28 22:04:35.776569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776511 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 22:04:35.776569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776544 2557 factory.go:103] Registering Raw factory Jul 28 22:04:35.776569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.776561 2557 manager.go:1196] Started watching for new ooms in manager Jul 28 22:04:35.777015 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.777000 2557 manager.go:349] Starting recovery of all containers Jul 28 22:04:35.779213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.779186 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:04:35.782036 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.781979 2557 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-225.ec2.internal\" not found" node="ip-10-0-142-225.ec2.internal" Jul 28 22:04:35.784250 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.784203 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 22:04:35.788713 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.788695 2557 manager.go:354] Recovery completed Jul 28 22:04:35.793705 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.793690 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:04:35.795611 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.795596 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:04:35.795679 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.795627 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:04:35.795679 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.795638 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:04:35.796241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.796228 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 22:04:35.796241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.796239 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 22:04:35.796347 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.796256 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 28 22:04:35.796347 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.796313 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:04:35.797142 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.797125 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:04:35.797227 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.797159 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:04:35.797227 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.797177 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:04:35.798434 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.798420 2557 policy_none.go:49] "None policy: Start" Jul 28 22:04:35.798495 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.798438 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 22:04:35.798495 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.798449 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 22:04:35.799502 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.799490 2557 policy_none.go:47] "Start" Jul 28 22:04:35.835892 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.835660 2557 manager.go:337] "Starting Device Plugin manager" Jul 28 22:04:35.836057 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.835928 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 22:04:35.836057 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.835943 2557 server.go:87] "Starting device plugin registration server" Jul 28 22:04:35.836256 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.836242 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 22:04:35.836297 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.836258 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 22:04:35.836400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.836381 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 22:04:35.836495 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.836479 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 22:04:35.836495 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.836490 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 22:04:35.837147 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.837122 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 22:04:35.837280 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.837174 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:35.937527 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.937426 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:04:35.939574 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.939151 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:04:35.939574 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.939185 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:04:35.939574 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.939200 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:04:35.939574 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.939239 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-225.ec2.internal" Jul 28 22:04:35.952932 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:35.952903 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-225.ec2.internal" Jul 28 22:04:35.953050 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.952956 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-225.ec2.internal\": node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:35.968139 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:35.968114 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:36.039055 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.039020 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 22:04:36.039237 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.039068 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 22:04:36.039237 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.039093 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 22:04:36.039237 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.039131 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 22:04:36.042238 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.042216 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:04:36.069574 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.069536 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:36.140252 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.140207 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal"] Jul 28 22:04:36.140335 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.140306 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:04:36.142016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.141991 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:04:36.142108 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.142020 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:04:36.142108 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.142031 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:04:36.143506 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.143491 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:04:36.143646 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.143629 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.143708 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.143671 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:04:36.144299 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.144282 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:04:36.144357 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.144310 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:04:36.144357 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.144320 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:04:36.144427 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.144284 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:04:36.144427 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.144393 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:04:36.144427 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.144405 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:04:36.145627 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.145613 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.145682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.145636 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:04:36.146384 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.146368 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:04:36.146495 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.146395 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:04:36.146495 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.146409 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:04:36.169777 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.169747 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:36.169887 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.169856 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-225.ec2.internal\" not found" node="ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.174695 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.174680 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-225.ec2.internal\" not found" node="ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.179208 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.179190 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0269d25d27c08c2cc14488ccb451dba5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-225.ec2.internal\" (UID: \"0269d25d27c08c2cc14488ccb451dba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.179270 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.179217 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cb85e1441cf636ff6c526ecb5a002c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal\" (UID: \"4cb85e1441cf636ff6c526ecb5a002c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.179270 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.179238 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cb85e1441cf636ff6c526ecb5a002c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal\" (UID: \"4cb85e1441cf636ff6c526ecb5a002c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.270100 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.270061 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:36.280170 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.280141 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0269d25d27c08c2cc14488ccb451dba5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-225.ec2.internal\" (UID: \"0269d25d27c08c2cc14488ccb451dba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.280239 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.280174 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cb85e1441cf636ff6c526ecb5a002c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal\" (UID: \"4cb85e1441cf636ff6c526ecb5a002c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.280239 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.280210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cb85e1441cf636ff6c526ecb5a002c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal\" (UID: \"4cb85e1441cf636ff6c526ecb5a002c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.280325 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.280253 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cb85e1441cf636ff6c526ecb5a002c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal\" (UID: \"4cb85e1441cf636ff6c526ecb5a002c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.280325 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.280257 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0269d25d27c08c2cc14488ccb451dba5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-225.ec2.internal\" (UID: \"0269d25d27c08c2cc14488ccb451dba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.280325 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.280273 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cb85e1441cf636ff6c526ecb5a002c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal\" (UID: \"4cb85e1441cf636ff6c526ecb5a002c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.376742 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.376696 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:36.479936 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.479895 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:36.480057 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.479964 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.480057 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.479989 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.580810 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.580700 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-225.ec2.internal\" not found" Jul 28 22:04:36.635594 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.635569 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:04:36.678203 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.678171 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.693546 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.693514 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 22:04:36.695574 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.695553 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.699775 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.699749 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 22:04:36.699899 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.699872 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 22:04:36.699990 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.699940 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 22:04:36.699990 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.699950 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 22:04:36.699990 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.699931 2557 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a0739143a033b41c9a17fb5f30212ba4-e870238e50cda619.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.142.225:60494->3.212.128.76:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" Jul 28 22:04:36.745104 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.745066 2557 apiserver.go:52] "Watching apiserver" Jul 28 22:04:36.749803 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.749783 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:04:36.753764 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.753745 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 22:04:36.755205 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.755182 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-2kljv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal","openshift-multus/multus-additional-cni-plugins-b5qcq","openshift-multus/network-metrics-daemon-vg2d2","openshift-ovn-kubernetes/ovnkube-node-wtzj4","kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb","openshift-dns/node-resolver-65mzx","openshift-image-registry/node-ca-5zjn6","openshift-multus/multus-slckt","openshift-network-diagnostics/network-check-target-g6fnc","openshift-network-operator/iptables-alerter-tmrzl","kube-system/global-pull-secret-syncer-76hsd","kube-system/konnectivity-agent-9jwvn"] Jul 28 22:04:36.756781 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.756765 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.758020 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.757987 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:36.758105 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.758049 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:36.759301 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.759280 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 22:04:36.759730 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.759703 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 22:04:36.759730 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.759732 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p794c\"" Jul 28 22:04:36.761176 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.759833 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.761176 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.759983 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 22:04:36.761915 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.761892 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:36.762178 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.762148 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:36.762765 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.762720 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zclqn\"" Jul 28 22:04:36.762765 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.762738 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 22:04:36.762946 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.762781 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 22:04:36.762946 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.762740 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 22:04:36.762946 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.762839 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 22:04:36.763386 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.763370 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 22:04:36.763630 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.763615 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.765038 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.765020 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:04:36.766371 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.766351 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.766779 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.766762 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 22:04:36.766830 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.766797 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lsxdk\"" Jul 28 22:04:36.767182 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.767167 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 22:04:36.767236 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.767184 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 22:04:36.767236 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.767175 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 22:04:36.767678 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.767662 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.767719 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.767677 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 22:04:36.768744 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.768724 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8c8hf\"" Jul 28 22:04:36.768744 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.768734 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 22:04:36.768903 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.768734 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 22:04:36.768903 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.768798 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 22:04:36.769033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.768993 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.769157 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.769139 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 22:04:36.769246 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.769227 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 22:04:36.769443 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.769428 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-499ng\"" Jul 28 22:04:36.770270 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.770254 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 22:04:36.770535 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.770519 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-slckt" Jul 28 22:04:36.770674 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.770613 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 21:59:36 +0000 UTC" deadline="2028-02-25 09:26:16.732244858 +0000 UTC" Jul 28 22:04:36.770740 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.770683 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13835h21m39.961565563s" Jul 28 22:04:36.771229 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.771213 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 22:04:36.771742 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.771729 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:04:36.771809 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.771769 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7wbrc\"" Jul 28 22:04:36.771864 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.771815 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 22:04:36.771904 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.771886 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:04:36.772161 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.772143 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:36.772237 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.772217 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:36.772289 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.772249 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wdbc4\"" Jul 28 22:04:36.772993 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.772975 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p7r8b\"" Jul 28 22:04:36.773121 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.772981 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 22:04:36.773350 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.773334 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.775235 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.775217 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 22:04:36.776155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.775850 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 22:04:36.776155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.775960 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 22:04:36.776155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.775970 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 22:04:36.776155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.776054 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-95vz5\"" Jul 28 22:04:36.776979 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.776964 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 22:04:36.782094 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782075 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-run-netns\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.782175 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782100 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1232cacc-c457-432f-99fc-90fcd75903a5-ovnkube-config\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.782175 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782118 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57-hosts-file\") pod \"node-resolver-65mzx\" (UID: \"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57\") " pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.782175 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782134 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-var-lib-cni-multus\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.782175 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782152 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-os-release\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.782175 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782170 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-systemd\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.782400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782207 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-socket-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.782400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782252 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-run-systemd\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.782400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782274 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-system-cni-dir\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.782400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782289 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-run-netns\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.782400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782309 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-sysconfig\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.782400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782324 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-run\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.782400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782348 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1232cacc-c457-432f-99fc-90fcd75903a5-env-overrides\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.782400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782390 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-multus-socket-dir-parent\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782423 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-run-multus-certs\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782459 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-sysctl-conf\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782483 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhj8j\" (UniqueName: \"kubernetes.io/projected/4f7813ac-e730-4a92-8bc9-121af675b07d-kube-api-access-nhj8j\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782508 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-kubelet\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782537 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-node-log\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782557 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-run-k8s-cni-cncf-io\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782583 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-var-lib-kubelet\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782609 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-hostroot\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782630 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdn9\" (UniqueName: \"kubernetes.io/projected/f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470-kube-api-access-hrdn9\") pod \"node-ca-5zjn6\" (UID: \"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470\") " pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.782689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782667 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5137a1e9-ebea-49c3-9d66-9f836d3bb074-cni-binary-copy\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782702 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5137a1e9-ebea-49c3-9d66-9f836d3bb074-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782734 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-cni-netd\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782761 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-system-cni-dir\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782784 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trbj\" (UniqueName: \"kubernetes.io/projected/5137a1e9-ebea-49c3-9d66-9f836d3bb074-kube-api-access-9trbj\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782839 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f659954-188f-4265-a463-cfd4ede0baa7-host-slash\") pod \"iptables-alerter-tmrzl\" (UID: \"9f659954-188f-4265-a463-cfd4ede0baa7\") " pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782865 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-host\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782896 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-slash\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.782956 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/602706ce-d40c-4ee5-9918-4f9855f11cb4-agent-certs\") pod \"konnectivity-agent-9jwvn\" (UID: \"602706ce-d40c-4ee5-9918-4f9855f11cb4\") " pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783009 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-etc-kubernetes\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.783076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783051 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vz7w\" (UniqueName: \"kubernetes.io/projected/95a72516-dfc4-450e-a5db-a3148727c75a-kube-api-access-2vz7w\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783083 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-sysctl-d\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783136 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-sys\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783173 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-log-socket\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783213 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-cnibin\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783242 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470-serviceca\") pod \"node-ca-5zjn6\" (UID: \"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470\") " pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783267 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783290 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f659954-188f-4265-a463-cfd4ede0baa7-iptables-alerter-script\") pod \"iptables-alerter-tmrzl\" (UID: \"9f659954-188f-4265-a463-cfd4ede0baa7\") " pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783309 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-device-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783323 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-cni-bin\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783342 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-485rx\" (UniqueName: \"kubernetes.io/projected/c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57-kube-api-access-485rx\") pod \"node-resolver-65mzx\" (UID: \"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57\") " pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783371 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783398 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c73c16b-804e-403e-924e-a320ab5cff73-etc-tuned\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783427 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-etc-openvswitch\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783448 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-run-ovn\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783468 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470-host\") pod \"node-ca-5zjn6\" (UID: \"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470\") " pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.783517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783494 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-modprobe-d\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783573 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-var-lib-kubelet\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783601 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1232cacc-c457-432f-99fc-90fcd75903a5-ovnkube-script-lib\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783626 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5137a1e9-ebea-49c3-9d66-9f836d3bb074-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783663 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-sys-fs\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783735 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783758 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-os-release\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783821 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-var-lib-cni-bin\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783879 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-cnibin\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783903 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-lib-modules\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.783996 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7b82635-0361-44b9-9b97-7f496d2be84c-kubelet-config\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784012 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784026 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1232cacc-c457-432f-99fc-90fcd75903a5-ovn-node-metrics-cert\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784040 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95a72516-dfc4-450e-a5db-a3148727c75a-multus-daemon-config\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784055 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7d7\" (UniqueName: \"kubernetes.io/projected/1c73c16b-804e-403e-924e-a320ab5cff73-kube-api-access-zz7d7\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784099 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-run-openvswitch\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.784241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784136 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784156 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57-tmp-dir\") pod \"node-resolver-65mzx\" (UID: \"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57\") " pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784172 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784188 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784209 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c73c16b-804e-403e-924e-a320ab5cff73-tmp\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784224 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-systemd-units\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784244 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwp6\" (UniqueName: \"kubernetes.io/projected/1232cacc-c457-432f-99fc-90fcd75903a5-kube-api-access-2lwp6\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784285 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-multus-cni-dir\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784326 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95a72516-dfc4-450e-a5db-a3148727c75a-cni-binary-copy\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784350 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-multus-conf-dir\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784366 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8kcp\" (UniqueName: \"kubernetes.io/projected/81accbfc-3d54-4286-87ff-3a6a34407c5d-kube-api-access-x8kcp\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784385 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-kubernetes\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784407 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-registration-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784425 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-var-lib-openvswitch\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784440 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/602706ce-d40c-4ee5-9918-4f9855f11cb4-konnectivity-ca\") pod \"konnectivity-agent-9jwvn\" (UID: \"602706ce-d40c-4ee5-9918-4f9855f11cb4\") " pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784454 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:36.784718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkml\" (UniqueName: \"kubernetes.io/projected/9f659954-188f-4265-a463-cfd4ede0baa7-kube-api-access-xzkml\") pod \"iptables-alerter-tmrzl\" (UID: \"9f659954-188f-4265-a463-cfd4ede0baa7\") " pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.785256 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.784484 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7b82635-0361-44b9-9b97-7f496d2be84c-dbus\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:36.788468 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.788446 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 22:04:36.816859 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.816839 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nhkf5" Jul 28 22:04:36.831194 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.831131 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nhkf5" Jul 28 22:04:36.884690 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884659 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1232cacc-c457-432f-99fc-90fcd75903a5-env-overrides\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.884867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884703 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-multus-socket-dir-parent\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.884867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884730 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-run-multus-certs\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.884867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884753 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-sysctl-conf\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.884867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884777 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhj8j\" (UniqueName: \"kubernetes.io/projected/4f7813ac-e730-4a92-8bc9-121af675b07d-kube-api-access-nhj8j\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.884867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884805 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-kubelet\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.884867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884817 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-run-multus-certs\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.884867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884829 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-node-log\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884855 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-run-k8s-cni-cncf-io\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884903 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-multus-socket-dir-parent\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884912 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-kubelet\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884907 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-var-lib-kubelet\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884984 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-var-lib-kubelet\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884996 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-node-log\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.884986 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-hostroot\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885032 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-run-k8s-cni-cncf-io\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885035 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-sysctl-conf\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885043 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdn9\" (UniqueName: \"kubernetes.io/projected/f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470-kube-api-access-hrdn9\") pod \"node-ca-5zjn6\" (UID: \"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470\") " pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885074 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-hostroot\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885096 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5137a1e9-ebea-49c3-9d66-9f836d3bb074-cni-binary-copy\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885126 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5137a1e9-ebea-49c3-9d66-9f836d3bb074-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885167 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-cni-netd\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885195 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-system-cni-dir\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885222 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-cni-netd\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.885251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885222 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trbj\" (UniqueName: \"kubernetes.io/projected/5137a1e9-ebea-49c3-9d66-9f836d3bb074-kube-api-access-9trbj\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885241 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1232cacc-c457-432f-99fc-90fcd75903a5-env-overrides\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885255 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-system-cni-dir\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885266 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f659954-188f-4265-a463-cfd4ede0baa7-host-slash\") pod \"iptables-alerter-tmrzl\" (UID: \"9f659954-188f-4265-a463-cfd4ede0baa7\") " pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-host\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885312 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f659954-188f-4265-a463-cfd4ede0baa7-host-slash\") pod \"iptables-alerter-tmrzl\" (UID: \"9f659954-188f-4265-a463-cfd4ede0baa7\") " pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885332 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-slash\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885356 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/602706ce-d40c-4ee5-9918-4f9855f11cb4-agent-certs\") pod \"konnectivity-agent-9jwvn\" (UID: \"602706ce-d40c-4ee5-9918-4f9855f11cb4\") " pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885364 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-host\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885382 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-etc-kubernetes\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885410 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vz7w\" (UniqueName: \"kubernetes.io/projected/95a72516-dfc4-450e-a5db-a3148727c75a-kube-api-access-2vz7w\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885439 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-sysctl-d\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885437 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-slash\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885462 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-sys\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885472 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-etc-kubernetes\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885484 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-log-socket\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885576 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-sys\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885609 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-cnibin\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.886155 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885634 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470-serviceca\") pod \"node-ca-5zjn6\" (UID: \"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470\") " pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885646 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-sysctl-d\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885659 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885684 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f659954-188f-4265-a463-cfd4ede0baa7-iptables-alerter-script\") pod \"iptables-alerter-tmrzl\" (UID: \"9f659954-188f-4265-a463-cfd4ede0baa7\") " pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885701 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-cnibin\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885709 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-device-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885744 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-log-socket\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885751 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-cni-bin\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885754 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885771 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5137a1e9-ebea-49c3-9d66-9f836d3bb074-cni-binary-copy\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885771 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5137a1e9-ebea-49c3-9d66-9f836d3bb074-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-485rx\" (UniqueName: \"kubernetes.io/projected/c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57-kube-api-access-485rx\") pod \"node-resolver-65mzx\" (UID: \"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57\") " pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885830 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-cni-bin\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885840 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885850 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.885928 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-device-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c73c16b-804e-403e-924e-a320ab5cff73-etc-tuned\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.887704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886088 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-etc-openvswitch\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886127 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-run-ovn\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470-host\") pod \"node-ca-5zjn6\" (UID: \"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470\") " pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886192 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-etc-openvswitch\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886208 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470-host\") pod \"node-ca-5zjn6\" (UID: \"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470\") " pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886224 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-modprobe-d\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886247 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-var-lib-kubelet\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886255 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-run-ovn\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1232cacc-c457-432f-99fc-90fcd75903a5-ovnkube-script-lib\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5137a1e9-ebea-49c3-9d66-9f836d3bb074-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886316 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-sys-fs\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886337 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886344 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470-serviceca\") pod \"node-ca-5zjn6\" (UID: \"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470\") " pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886353 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-modprobe-d\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886358 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-os-release\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886418 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-var-lib-cni-bin\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886429 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f659954-188f-4265-a463-cfd4ede0baa7-iptables-alerter-script\") pod \"iptables-alerter-tmrzl\" (UID: \"9f659954-188f-4265-a463-cfd4ede0baa7\") " pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886440 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-cnibin\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.888685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886467 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-lib-modules\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886492 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7b82635-0361-44b9-9b97-7f496d2be84c-kubelet-config\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886505 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-os-release\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886560 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7b82635-0361-44b9-9b97-7f496d2be84c-kubelet-config\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886569 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-sys-fs\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886598 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886629 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1232cacc-c457-432f-99fc-90fcd75903a5-ovn-node-metrics-cert\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886656 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95a72516-dfc4-450e-a5db-a3148727c75a-multus-daemon-config\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886683 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7d7\" (UniqueName: \"kubernetes.io/projected/1c73c16b-804e-403e-924e-a320ab5cff73-kube-api-access-zz7d7\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886750 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-run-openvswitch\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886782 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886810 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57-tmp-dir\") pod \"node-resolver-65mzx\" (UID: \"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57\") " pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886837 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886866 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.886889 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c73c16b-804e-403e-924e-a320ab5cff73-tmp\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887005 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-systemd-units\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.889564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887060 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1232cacc-c457-432f-99fc-90fcd75903a5-ovnkube-script-lib\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887087 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwp6\" (UniqueName: \"kubernetes.io/projected/1232cacc-c457-432f-99fc-90fcd75903a5-kube-api-access-2lwp6\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-multus-cni-dir\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887140 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-cnibin\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887144 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95a72516-dfc4-450e-a5db-a3148727c75a-cni-binary-copy\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887178 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-multus-conf-dir\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887202 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kcp\" (UniqueName: \"kubernetes.io/projected/81accbfc-3d54-4286-87ff-3a6a34407c5d-kube-api-access-x8kcp\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887115 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-var-lib-cni-bin\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887284 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-kubernetes\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-registration-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887356 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-var-lib-openvswitch\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/602706ce-d40c-4ee5-9918-4f9855f11cb4-konnectivity-ca\") pod \"konnectivity-agent-9jwvn\" (UID: \"602706ce-d40c-4ee5-9918-4f9855f11cb4\") " pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887406 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887432 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkml\" (UniqueName: \"kubernetes.io/projected/9f659954-188f-4265-a463-cfd4ede0baa7-kube-api-access-xzkml\") pod \"iptables-alerter-tmrzl\" (UID: \"9f659954-188f-4265-a463-cfd4ede0baa7\") " pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887458 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7b82635-0361-44b9-9b97-7f496d2be84c-dbus\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887478 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-lib-modules\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887487 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-run-netns\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.891076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887518 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1232cacc-c457-432f-99fc-90fcd75903a5-ovnkube-config\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887553 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-run-openvswitch\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.887816 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.888117 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/602706ce-d40c-4ee5-9918-4f9855f11cb4-konnectivity-ca\") pod \"konnectivity-agent-9jwvn\" (UID: \"602706ce-d40c-4ee5-9918-4f9855f11cb4\") " pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.888248 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.888316 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs podName:81accbfc-3d54-4286-87ff-3a6a34407c5d nodeName:}" failed. No retries permitted until 2026-07-28 22:04:37.388294305 +0000 UTC m=+2.026153419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs") pod "network-metrics-daemon-vg2d2" (UID: "81accbfc-3d54-4286-87ff-3a6a34407c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.888685 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7b82635-0361-44b9-9b97-7f496d2be84c-dbus\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.888738 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-run-netns\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.888783 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57-tmp-dir\") pod \"node-resolver-65mzx\" (UID: \"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57\") " pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.888847 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.888897 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret podName:e7b82635-0361-44b9-9b97-7f496d2be84c nodeName:}" failed. No retries permitted until 2026-07-28 22:04:37.388879847 +0000 UTC m=+2.026738956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret") pod "global-pull-secret-syncer-76hsd" (UID: "e7b82635-0361-44b9-9b97-7f496d2be84c") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.888845 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.888922 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.889173 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-multus-conf-dir\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.889529 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-var-lib-kubelet\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.889880 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57-hosts-file\") pod \"node-resolver-65mzx\" (UID: \"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57\") " pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.889941 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-multus-cni-dir\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.891626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.889957 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-var-lib-cni-multus\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.889969 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c73c16b-804e-403e-924e-a320ab5cff73-tmp\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.889885 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-registration-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.889989 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-os-release\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890039 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-kubernetes\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890110 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5137a1e9-ebea-49c3-9d66-9f836d3bb074-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890134 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-var-lib-openvswitch\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890187 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5137a1e9-ebea-49c3-9d66-9f836d3bb074-os-release\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890211 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-var-lib-cni-multus\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890222 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-systemd\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890254 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-socket-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890276 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57-hosts-file\") pod \"node-resolver-65mzx\" (UID: \"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57\") " pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890282 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-run-systemd\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890281 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1232cacc-c457-432f-99fc-90fcd75903a5-ovnkube-config\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890328 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-run-systemd\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890346 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1232cacc-c457-432f-99fc-90fcd75903a5-systemd-units\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890347 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-systemd\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.892418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890354 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1232cacc-c457-432f-99fc-90fcd75903a5-ovn-node-metrics-cert\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-system-cni-dir\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890405 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/602706ce-d40c-4ee5-9918-4f9855f11cb4-agent-certs\") pod \"konnectivity-agent-9jwvn\" (UID: \"602706ce-d40c-4ee5-9918-4f9855f11cb4\") " pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890412 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-system-cni-dir\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890405 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-run-netns\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890440 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95a72516-dfc4-450e-a5db-a3148727c75a-host-run-netns\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890444 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f7813ac-e730-4a92-8bc9-121af675b07d-socket-dir\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890454 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-sysconfig\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890472 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95a72516-dfc4-450e-a5db-a3148727c75a-cni-binary-copy\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890490 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-etc-sysconfig\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890491 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-run\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890522 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95a72516-dfc4-450e-a5db-a3148727c75a-multus-daemon-config\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890530 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c73c16b-804e-403e-924e-a320ab5cff73-run\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.892991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.890603 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c73c16b-804e-403e-924e-a320ab5cff73-etc-tuned\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:36.894161 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.894143 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdn9\" (UniqueName: \"kubernetes.io/projected/f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470-kube-api-access-hrdn9\") pod \"node-ca-5zjn6\" (UID: \"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470\") " pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.894554 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.894536 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhj8j\" (UniqueName: \"kubernetes.io/projected/4f7813ac-e730-4a92-8bc9-121af675b07d-kube-api-access-nhj8j\") pod \"aws-ebs-csi-driver-node-bgsfb\" (UID: \"4f7813ac-e730-4a92-8bc9-121af675b07d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:36.894598 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.894583 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vz7w\" (UniqueName: \"kubernetes.io/projected/95a72516-dfc4-450e-a5db-a3148727c75a-kube-api-access-2vz7w\") pod \"multus-slckt\" (UID: \"95a72516-dfc4-450e-a5db-a3148727c75a\") " pod="openshift-multus/multus-slckt" Jul 28 22:04:36.900638 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.900616 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:04:36.900638 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.900648 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:04:36.900808 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.900662 2557 projected.go:196] Error preparing data for projected volume kube-api-access-jvsjg for pod openshift-network-diagnostics/network-check-target-g6fnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:36.900808 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:36.900780 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg podName:b0c0f55b-3d81-4bf6-8085-5e2a335e5799 nodeName:}" failed. No retries permitted until 2026-07-28 22:04:37.4007619 +0000 UTC m=+2.038621028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jvsjg" (UniqueName: "kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg") pod "network-check-target-g6fnc" (UID: "b0c0f55b-3d81-4bf6-8085-5e2a335e5799") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:36.903054 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.903030 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-485rx\" (UniqueName: \"kubernetes.io/projected/c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57-kube-api-access-485rx\") pod \"node-resolver-65mzx\" (UID: \"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57\") " pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:36.903282 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.903224 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trbj\" (UniqueName: \"kubernetes.io/projected/5137a1e9-ebea-49c3-9d66-9f836d3bb074-kube-api-access-9trbj\") pod \"multus-additional-cni-plugins-b5qcq\" (UID: \"5137a1e9-ebea-49c3-9d66-9f836d3bb074\") " pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:36.903282 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.903244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkml\" (UniqueName: \"kubernetes.io/projected/9f659954-188f-4265-a463-cfd4ede0baa7-kube-api-access-xzkml\") pod \"iptables-alerter-tmrzl\" (UID: \"9f659954-188f-4265-a463-cfd4ede0baa7\") " pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:36.903572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.903544 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5zjn6" Jul 28 22:04:36.903572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.903554 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwp6\" (UniqueName: \"kubernetes.io/projected/1232cacc-c457-432f-99fc-90fcd75903a5-kube-api-access-2lwp6\") pod \"ovnkube-node-wtzj4\" (UID: \"1232cacc-c457-432f-99fc-90fcd75903a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:36.904591 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.904572 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kcp\" (UniqueName: \"kubernetes.io/projected/81accbfc-3d54-4286-87ff-3a6a34407c5d-kube-api-access-x8kcp\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:36.905262 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:36.905244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7d7\" (UniqueName: \"kubernetes.io/projected/1c73c16b-804e-403e-924e-a320ab5cff73-kube-api-access-zz7d7\") pod \"tuned-2kljv\" (UID: \"1c73c16b-804e-403e-924e-a320ab5cff73\") " pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:37.075966 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.075932 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0269d25d27c08c2cc14488ccb451dba5.slice/crio-481d534082304de3bd42899703aaac3f3991faa26c5ee1da710cf4df3f2deea1 WatchSource:0}: Error finding container 481d534082304de3bd42899703aaac3f3991faa26c5ee1da710cf4df3f2deea1: Status 404 returned error can't find the container with id 481d534082304de3bd42899703aaac3f3991faa26c5ee1da710cf4df3f2deea1 Jul 28 22:04:37.076575 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.076550 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb85e1441cf636ff6c526ecb5a002c1.slice/crio-498f1ab009b2e0ab03c58cc580c40e71454cb287bd78bb1b5ba3818ff34f07fb WatchSource:0}: Error finding container 498f1ab009b2e0ab03c58cc580c40e71454cb287bd78bb1b5ba3818ff34f07fb: Status 404 returned error can't find the container with id 498f1ab009b2e0ab03c58cc580c40e71454cb287bd78bb1b5ba3818ff34f07fb Jul 28 22:04:37.082758 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.082741 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:04:37.089494 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.089473 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" Jul 28 22:04:37.095292 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.095271 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" Jul 28 22:04:37.097586 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.097557 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7813ac_e730_4a92_8bc9_121af675b07d.slice/crio-22ced125541305f87ae058e2f7c48c6e32d2106e5223f7325d4882f14497cd5f WatchSource:0}: Error finding container 22ced125541305f87ae058e2f7c48c6e32d2106e5223f7325d4882f14497cd5f: Status 404 returned error can't find the container with id 22ced125541305f87ae058e2f7c48c6e32d2106e5223f7325d4882f14497cd5f Jul 28 22:04:37.103822 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.103800 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:04:37.103996 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.103821 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5137a1e9_ebea_49c3_9d66_9f836d3bb074.slice/crio-1a19948fc3442d5bace12739d5fdc092289c5cb6419ffd649d14a52b06504d42 WatchSource:0}: Error finding container 1a19948fc3442d5bace12739d5fdc092289c5cb6419ffd649d14a52b06504d42: Status 404 returned error can't find the container with id 1a19948fc3442d5bace12739d5fdc092289c5cb6419ffd649d14a52b06504d42 Jul 28 22:04:37.112008 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.111967 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1232cacc_c457_432f_99fc_90fcd75903a5.slice/crio-8e9fad7e1bd9d7a7c5bf111161cbcbfc54b4a0892080ca32c909979d280282c6 WatchSource:0}: Error finding container 8e9fad7e1bd9d7a7c5bf111161cbcbfc54b4a0892080ca32c909979d280282c6: Status 404 returned error can't find the container with id 8e9fad7e1bd9d7a7c5bf111161cbcbfc54b4a0892080ca32c909979d280282c6 Jul 28 22:04:37.130941 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.130908 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:04:37.139716 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.139684 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602706ce_d40c_4ee5_9918_4f9855f11cb4.slice/crio-06a17d30787f470684ec1db724b59afa4acf0c52338930c5ffe0cd2e6c50d23d WatchSource:0}: Error finding container 06a17d30787f470684ec1db724b59afa4acf0c52338930c5ffe0cd2e6c50d23d: Status 404 returned error can't find the container with id 06a17d30787f470684ec1db724b59afa4acf0c52338930c5ffe0cd2e6c50d23d Jul 28 22:04:37.148862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.148838 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-65mzx" Jul 28 22:04:37.153427 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.153403 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tmrzl" Jul 28 22:04:37.157862 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.157839 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4009ba5_5c0a_4aa5_8ad6_f1b30c368c57.slice/crio-74b4ce7cf4229f0805ad3fcf254f683c1abe2fc9ebc07dd98817adf140b6c2cb WatchSource:0}: Error finding container 74b4ce7cf4229f0805ad3fcf254f683c1abe2fc9ebc07dd98817adf140b6c2cb: Status 404 returned error can't find the container with id 74b4ce7cf4229f0805ad3fcf254f683c1abe2fc9ebc07dd98817adf140b6c2cb Jul 28 22:04:37.162318 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.162292 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f659954_188f_4265_a463_cfd4ede0baa7.slice/crio-2fa547d5377ce9d442c0af99c88ccc64cdc8ca824841f40edecc09601c65a276 WatchSource:0}: Error finding container 2fa547d5377ce9d442c0af99c88ccc64cdc8ca824841f40edecc09601c65a276: Status 404 returned error can't find the container with id 2fa547d5377ce9d442c0af99c88ccc64cdc8ca824841f40edecc09601c65a276 Jul 28 22:04:37.167472 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.167455 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kljv" Jul 28 22:04:37.177259 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.177230 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c73c16b_804e_403e_924e_a320ab5cff73.slice/crio-8b96ae94ce06d966c6e67f4858a3de6213f64e9b9b2722c4fdecef8afb4c2d87 WatchSource:0}: Error finding container 8b96ae94ce06d966c6e67f4858a3de6213f64e9b9b2722c4fdecef8afb4c2d87: Status 404 returned error can't find the container with id 8b96ae94ce06d966c6e67f4858a3de6213f64e9b9b2722c4fdecef8afb4c2d87 Jul 28 22:04:37.191116 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.191095 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-slckt" Jul 28 22:04:37.199209 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:37.199183 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a72516_dfc4_450e_a5db_a3148727c75a.slice/crio-0b949bdcf2368952c602cc36f54493279d715fa7a6b54cdf4d1bda913d367c6f WatchSource:0}: Error finding container 0b949bdcf2368952c602cc36f54493279d715fa7a6b54cdf4d1bda913d367c6f: Status 404 returned error can't find the container with id 0b949bdcf2368952c602cc36f54493279d715fa7a6b54cdf4d1bda913d367c6f Jul 28 22:04:37.393500 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.393411 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:37.393500 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.393451 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:37.393735 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:37.393574 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:37.393735 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:37.393574 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:37.393735 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:37.393651 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret podName:e7b82635-0361-44b9-9b97-7f496d2be84c nodeName:}" failed. No retries permitted until 2026-07-28 22:04:38.393635131 +0000 UTC m=+3.031494244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret") pod "global-pull-secret-syncer-76hsd" (UID: "e7b82635-0361-44b9-9b97-7f496d2be84c") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:37.393735 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:37.393715 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs podName:81accbfc-3d54-4286-87ff-3a6a34407c5d nodeName:}" failed. No retries permitted until 2026-07-28 22:04:38.393688375 +0000 UTC m=+3.031547506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs") pod "network-metrics-daemon-vg2d2" (UID: "81accbfc-3d54-4286-87ff-3a6a34407c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:37.494195 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.494158 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:37.494393 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:37.494349 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:04:37.494393 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:37.494375 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:04:37.494393 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:37.494392 2557 projected.go:196] Error preparing data for projected volume kube-api-access-jvsjg for pod openshift-network-diagnostics/network-check-target-g6fnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:37.494516 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:37.494455 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg podName:b0c0f55b-3d81-4bf6-8085-5e2a335e5799 nodeName:}" failed. No retries permitted until 2026-07-28 22:04:38.494436049 +0000 UTC m=+3.132295159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvsjg" (UniqueName: "kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg") pod "network-check-target-g6fnc" (UID: "b0c0f55b-3d81-4bf6-8085-5e2a335e5799") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:37.826763 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.826729 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:04:37.833293 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.833230 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 21:59:37 +0000 UTC" deadline="2028-02-07 10:38:39.407629924 +0000 UTC" Jul 28 22:04:37.833293 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:37.833293 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13404h34m1.574340751s" Jul 28 22:04:38.042417 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.042385 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:38.042619 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.042514 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:38.042950 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.042930 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:38.043091 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.043053 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:38.056955 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.056833 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-slckt" event={"ID":"95a72516-dfc4-450e-a5db-a3148727c75a","Type":"ContainerStarted","Data":"0b949bdcf2368952c602cc36f54493279d715fa7a6b54cdf4d1bda913d367c6f"} Jul 28 22:04:38.066032 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.065821 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tmrzl" event={"ID":"9f659954-188f-4265-a463-cfd4ede0baa7","Type":"ContainerStarted","Data":"2fa547d5377ce9d442c0af99c88ccc64cdc8ca824841f40edecc09601c65a276"} Jul 28 22:04:38.068553 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.068514 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-65mzx" event={"ID":"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57","Type":"ContainerStarted","Data":"74b4ce7cf4229f0805ad3fcf254f683c1abe2fc9ebc07dd98817adf140b6c2cb"} Jul 28 22:04:38.086170 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.086080 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" event={"ID":"4f7813ac-e730-4a92-8bc9-121af675b07d","Type":"ContainerStarted","Data":"22ced125541305f87ae058e2f7c48c6e32d2106e5223f7325d4882f14497cd5f"} Jul 28 22:04:38.091404 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.091364 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" event={"ID":"0269d25d27c08c2cc14488ccb451dba5","Type":"ContainerStarted","Data":"481d534082304de3bd42899703aaac3f3991faa26c5ee1da710cf4df3f2deea1"} Jul 28 22:04:38.095170 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.095141 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kljv" event={"ID":"1c73c16b-804e-403e-924e-a320ab5cff73","Type":"ContainerStarted","Data":"8b96ae94ce06d966c6e67f4858a3de6213f64e9b9b2722c4fdecef8afb4c2d87"} Jul 28 22:04:38.109378 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.109340 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9jwvn" event={"ID":"602706ce-d40c-4ee5-9918-4f9855f11cb4","Type":"ContainerStarted","Data":"06a17d30787f470684ec1db724b59afa4acf0c52338930c5ffe0cd2e6c50d23d"} Jul 28 22:04:38.127579 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.127543 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" event={"ID":"1232cacc-c457-432f-99fc-90fcd75903a5","Type":"ContainerStarted","Data":"8e9fad7e1bd9d7a7c5bf111161cbcbfc54b4a0892080ca32c909979d280282c6"} Jul 28 22:04:38.140893 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.140862 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" event={"ID":"5137a1e9-ebea-49c3-9d66-9f836d3bb074","Type":"ContainerStarted","Data":"1a19948fc3442d5bace12739d5fdc092289c5cb6419ffd649d14a52b06504d42"} Jul 28 22:04:38.147029 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.146967 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" event={"ID":"4cb85e1441cf636ff6c526ecb5a002c1","Type":"ContainerStarted","Data":"498f1ab009b2e0ab03c58cc580c40e71454cb287bd78bb1b5ba3818ff34f07fb"} Jul 28 22:04:38.177131 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.177098 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:04:38.402883 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.402791 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:38.403233 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.402898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:38.403233 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.403041 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:38.403233 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.403104 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:38.403233 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.403116 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret podName:e7b82635-0361-44b9-9b97-7f496d2be84c nodeName:}" failed. No retries permitted until 2026-07-28 22:04:40.40309658 +0000 UTC m=+5.040955707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret") pod "global-pull-secret-syncer-76hsd" (UID: "e7b82635-0361-44b9-9b97-7f496d2be84c") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:38.403233 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.403147 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs podName:81accbfc-3d54-4286-87ff-3a6a34407c5d nodeName:}" failed. No retries permitted until 2026-07-28 22:04:40.403133989 +0000 UTC m=+5.040993108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs") pod "network-metrics-daemon-vg2d2" (UID: "81accbfc-3d54-4286-87ff-3a6a34407c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:38.503887 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.503843 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:38.504123 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.504105 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:04:38.504204 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.504132 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:04:38.504204 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.504145 2557 projected.go:196] Error preparing data for projected volume kube-api-access-jvsjg for pod openshift-network-diagnostics/network-check-target-g6fnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:38.504311 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:38.504209 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg podName:b0c0f55b-3d81-4bf6-8085-5e2a335e5799 nodeName:}" failed. No retries permitted until 2026-07-28 22:04:40.504189411 +0000 UTC m=+5.142048570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvsjg" (UniqueName: "kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg") pod "network-check-target-g6fnc" (UID: "b0c0f55b-3d81-4bf6-8085-5e2a335e5799") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:38.610044 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.605450 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:04:38.610219 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:04:38.610121 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e5aa99_abf9_4ac5_86a9_a34dc7d6f470.slice/crio-3fdc1d811b5944dadfcf24985267dacd54cb2a98e087598645d9018a5097c8ff WatchSource:0}: Error finding container 3fdc1d811b5944dadfcf24985267dacd54cb2a98e087598645d9018a5097c8ff: Status 404 returned error can't find the container with id 3fdc1d811b5944dadfcf24985267dacd54cb2a98e087598645d9018a5097c8ff Jul 28 22:04:38.834166 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.834119 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 21:59:37 +0000 UTC" deadline="2028-03-11 08:47:14.747504708 +0000 UTC" Jul 28 22:04:38.834166 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:38.834164 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14194h42m35.913345754s" Jul 28 22:04:39.040685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:39.040140 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:39.040685 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:39.040286 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:39.170510 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:39.170417 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5zjn6" event={"ID":"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470","Type":"ContainerStarted","Data":"3fdc1d811b5944dadfcf24985267dacd54cb2a98e087598645d9018a5097c8ff"} Jul 28 22:04:40.042380 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:40.042348 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:40.042888 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.042504 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:40.042888 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:40.042570 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:40.042888 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.042653 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:40.420605 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:40.420513 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:40.420605 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:40.420578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:40.420835 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.420725 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:40.420835 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.420788 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs podName:81accbfc-3d54-4286-87ff-3a6a34407c5d nodeName:}" failed. No retries permitted until 2026-07-28 22:04:44.420767013 +0000 UTC m=+9.058626128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs") pod "network-metrics-daemon-vg2d2" (UID: "81accbfc-3d54-4286-87ff-3a6a34407c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:40.421306 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.421284 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:40.421406 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.421345 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret podName:e7b82635-0361-44b9-9b97-7f496d2be84c nodeName:}" failed. No retries permitted until 2026-07-28 22:04:44.421330018 +0000 UTC m=+9.059189132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret") pod "global-pull-secret-syncer-76hsd" (UID: "e7b82635-0361-44b9-9b97-7f496d2be84c") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:40.521488 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:40.521454 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:40.521706 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.521681 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:04:40.521706 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.521704 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:04:40.521829 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.521718 2557 projected.go:196] Error preparing data for projected volume kube-api-access-jvsjg for pod openshift-network-diagnostics/network-check-target-g6fnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:40.521829 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:40.521776 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg podName:b0c0f55b-3d81-4bf6-8085-5e2a335e5799 nodeName:}" failed. No retries permitted until 2026-07-28 22:04:44.521758022 +0000 UTC m=+9.159617134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvsjg" (UniqueName: "kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg") pod "network-check-target-g6fnc" (UID: "b0c0f55b-3d81-4bf6-8085-5e2a335e5799") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:41.040768 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:41.040213 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:41.040768 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:41.040350 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:42.041036 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:42.040119 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:42.041036 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:42.040266 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:42.041036 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:42.040734 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:42.041036 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:42.040856 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:43.039761 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:43.039721 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:43.039996 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:43.039883 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:44.040973 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:44.040198 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:44.040973 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.040338 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:44.040973 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:44.040801 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:44.040973 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.040919 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:44.456068 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:44.455959 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:44.456068 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:44.456032 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:44.456386 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.456110 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:44.456386 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.456165 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:44.456386 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.456184 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret podName:e7b82635-0361-44b9-9b97-7f496d2be84c nodeName:}" failed. No retries permitted until 2026-07-28 22:04:52.456162453 +0000 UTC m=+17.094021583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret") pod "global-pull-secret-syncer-76hsd" (UID: "e7b82635-0361-44b9-9b97-7f496d2be84c") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:44.456386 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.456213 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs podName:81accbfc-3d54-4286-87ff-3a6a34407c5d nodeName:}" failed. No retries permitted until 2026-07-28 22:04:52.456190862 +0000 UTC m=+17.094049981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs") pod "network-metrics-daemon-vg2d2" (UID: "81accbfc-3d54-4286-87ff-3a6a34407c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:44.557276 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:44.557235 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:44.557469 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.557414 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:04:44.557469 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.557439 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:04:44.557469 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.557452 2557 projected.go:196] Error preparing data for projected volume kube-api-access-jvsjg for pod openshift-network-diagnostics/network-check-target-g6fnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:44.557662 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:44.557519 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg podName:b0c0f55b-3d81-4bf6-8085-5e2a335e5799 nodeName:}" failed. No retries permitted until 2026-07-28 22:04:52.557499659 +0000 UTC m=+17.195358789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvsjg" (UniqueName: "kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg") pod "network-check-target-g6fnc" (UID: "b0c0f55b-3d81-4bf6-8085-5e2a335e5799") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:45.040346 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:45.040306 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:45.040535 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:45.040453 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:46.043485 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:46.043448 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:46.043883 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:46.043575 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:46.043883 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:46.043655 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:46.043883 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:46.043738 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:47.039919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:47.039880 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:47.040141 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:47.040044 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:48.042408 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:48.042376 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:48.042408 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:48.042410 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:48.042860 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:48.042500 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:48.042860 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:48.042624 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:49.040092 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:49.040058 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:49.040250 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:49.040158 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:50.040126 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:50.040091 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:50.040501 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:50.040226 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:50.040501 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:50.040280 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:50.040501 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:50.040391 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:51.039994 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:51.039954 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:51.040238 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:51.040070 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:52.040180 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:52.040147 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:52.040374 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:52.040188 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:52.040374 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.040285 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:52.040796 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.040392 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:52.521154 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:52.521117 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:52.521154 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:52.521163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:52.521422 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.521281 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:52.521422 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.521346 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs podName:81accbfc-3d54-4286-87ff-3a6a34407c5d nodeName:}" failed. No retries permitted until 2026-07-28 22:05:08.521327986 +0000 UTC m=+33.159187093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs") pod "network-metrics-daemon-vg2d2" (UID: "81accbfc-3d54-4286-87ff-3a6a34407c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:04:52.521422 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.521281 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:52.521422 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.521395 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret podName:e7b82635-0361-44b9-9b97-7f496d2be84c nodeName:}" failed. No retries permitted until 2026-07-28 22:05:08.521384706 +0000 UTC m=+33.159243817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret") pod "global-pull-secret-syncer-76hsd" (UID: "e7b82635-0361-44b9-9b97-7f496d2be84c") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:04:52.621742 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:52.621694 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:52.621949 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.621851 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:04:52.621949 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.621871 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:04:52.621949 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.621881 2557 projected.go:196] Error preparing data for projected volume kube-api-access-jvsjg for pod openshift-network-diagnostics/network-check-target-g6fnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:52.622107 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:52.621961 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg podName:b0c0f55b-3d81-4bf6-8085-5e2a335e5799 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:08.621940228 +0000 UTC m=+33.259799355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvsjg" (UniqueName: "kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg") pod "network-check-target-g6fnc" (UID: "b0c0f55b-3d81-4bf6-8085-5e2a335e5799") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:04:53.039742 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:53.039702 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:53.039949 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:53.039826 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:54.039806 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:54.039720 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:54.040255 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:54.039839 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:54.040255 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:54.039958 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:54.040255 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:54.040091 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:55.039571 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:55.039536 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:55.039736 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:55.039672 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:56.040318 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:56.040282 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:56.040784 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:56.040399 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:56.040784 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:56.040483 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:56.040784 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:56.040592 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:57.040227 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:57.040189 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:57.040410 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:57.040318 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:58.039513 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:58.039356 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:04:58.039613 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:58.039595 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:04:58.039867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:58.039422 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:04:58.039977 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:58.039944 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:04:58.230880 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:58.230241 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" event={"ID":"0269d25d27c08c2cc14488ccb451dba5","Type":"ContainerStarted","Data":"2400d83007eb17ccb24815ae80ce7fb67156401683382a0f44f7f445a17d68f3"} Jul 28 22:04:58.233757 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:58.233635 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kljv" event={"ID":"1c73c16b-804e-403e-924e-a320ab5cff73","Type":"ContainerStarted","Data":"8b5715292e5a1bcb06fdc76582853f6d8545c8e436b96a8f2c7b6bb302793c22"} Jul 28 22:04:58.253981 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:58.253911 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-slckt" event={"ID":"95a72516-dfc4-450e-a5db-a3148727c75a","Type":"ContainerStarted","Data":"1f86a010524fcbf39300b0fea2ec691a8e56e481601b98af228c9eabfa914f7e"} Jul 28 22:04:58.264292 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:58.264174 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2kljv" podStartSLOduration=1.569259599 podStartE2EDuration="22.264149729s" totalImagesPullingTime="20.69489013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:04:58.263681005 +0000 UTC m=+22.901540133" watchObservedRunningTime="2026-07-28 22:04:58.264149729 +0000 UTC m=+22.902008859" Jul 28 22:04:58.264815 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:58.264774 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-225.ec2.internal" podStartSLOduration=21.264761598 podStartE2EDuration="21.264761598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:04:58.247408239 +0000 UTC m=+22.885267368" watchObservedRunningTime="2026-07-28 22:04:58.264761598 +0000 UTC m=+22.902620728" Jul 28 22:04:58.286047 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:58.285790 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-slckt" podStartSLOduration=0.594965531 podStartE2EDuration="21.285770155s" totalImagesPullingTime="20.690804624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:04:58.285474336 +0000 UTC m=+22.923333466" watchObservedRunningTime="2026-07-28 22:04:58.285770155 +0000 UTC m=+22.923629285" Jul 28 22:04:59.040117 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.040082 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:04:59.040253 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:04:59.040221 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:04:59.258183 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.258147 2557 generic.go:356] "Generic (PLEG): container finished" podID="4cb85e1441cf636ff6c526ecb5a002c1" containerID="2141615c2292fc1aca143f944c064672fa6710695542c1aeb6ce79ecf162ef3d" exitCode=0 Jul 28 22:04:59.258879 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.258229 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" event={"ID":"4cb85e1441cf636ff6c526ecb5a002c1","Type":"ContainerDied","Data":"2141615c2292fc1aca143f944c064672fa6710695542c1aeb6ce79ecf162ef3d"} Jul 28 22:04:59.259760 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.259736 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-65mzx" event={"ID":"c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57","Type":"ContainerStarted","Data":"fd4423d892e67542e1ee17373920c2da4da3847504966b75207a70cba03a5f6c"} Jul 28 22:04:59.261626 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.261605 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" event={"ID":"4f7813ac-e730-4a92-8bc9-121af675b07d","Type":"ContainerStarted","Data":"d1f3cdc9ddf96ad6d31ed14693a1b18ef6c8302baeba550604a1f5995b51fec6"} Jul 28 22:04:59.263266 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.263244 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5zjn6" event={"ID":"f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470","Type":"ContainerStarted","Data":"364a319a0ad41e99962dcf572c7bd9c6bfdeb02f19ca05bef855504e1515d43a"} Jul 28 22:04:59.266783 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.266764 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9jwvn" event={"ID":"602706ce-d40c-4ee5-9918-4f9855f11cb4","Type":"ContainerStarted","Data":"c686e0fd6a0c9b8c0472fe2ce2798d657213d50ec94b1a04fd4b1bc740379f94"} Jul 28 22:04:59.269764 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.269746 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" event={"ID":"1232cacc-c457-432f-99fc-90fcd75903a5","Type":"ContainerStarted","Data":"0d874455647c98dd0105c042e3e133268d399087bbfd4029864bc587ce7d4a48"} Jul 28 22:04:59.269865 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.269810 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" event={"ID":"1232cacc-c457-432f-99fc-90fcd75903a5","Type":"ContainerStarted","Data":"01b20736bb6bd39b2bba81955b008febda5561881460cd207a9ca5c1f637d7f7"} Jul 28 22:04:59.269919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.269864 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" event={"ID":"1232cacc-c457-432f-99fc-90fcd75903a5","Type":"ContainerStarted","Data":"4ab736197f10bbb3012eddf2e4b6ccd9095b7937fbae243b0889a31f22a5aea3"} Jul 28 22:04:59.269976 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.269915 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" event={"ID":"1232cacc-c457-432f-99fc-90fcd75903a5","Type":"ContainerStarted","Data":"78d0f8b312916bbb9fa9b45b43e6cbde5b5a681f729a8efe66780b0490e5a8ce"} Jul 28 22:04:59.270019 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.269975 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" event={"ID":"1232cacc-c457-432f-99fc-90fcd75903a5","Type":"ContainerStarted","Data":"3ffc33b8ea728fd69237a56028e50b843159bf5b2ddd1b133417215c3cf747ed"} Jul 28 22:04:59.270019 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.270005 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" event={"ID":"1232cacc-c457-432f-99fc-90fcd75903a5","Type":"ContainerStarted","Data":"605b6516b50cb228ebfe894376bcac427521acc6de149532789421e183201345"} Jul 28 22:04:59.271423 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.271401 2557 generic.go:356] "Generic (PLEG): container finished" podID="5137a1e9-ebea-49c3-9d66-9f836d3bb074" containerID="e7bd43c9e61e1d119f6cfb6e8c852dcb9998b6fadf129efcb566562f6c42ffb1" exitCode=0 Jul 28 22:04:59.271858 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.271839 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" event={"ID":"5137a1e9-ebea-49c3-9d66-9f836d3bb074","Type":"ContainerDied","Data":"e7bd43c9e61e1d119f6cfb6e8c852dcb9998b6fadf129efcb566562f6c42ffb1"} Jul 28 22:04:59.295914 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.295848 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-65mzx" podStartSLOduration=2.609247532 podStartE2EDuration="23.295830266s" totalImagesPullingTime="20.686582734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:04:59.295464297 +0000 UTC m=+23.933323429" watchObservedRunningTime="2026-07-28 22:04:59.295830266 +0000 UTC m=+23.933689374" Jul 28 22:04:59.311008 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.310948 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5zjn6" podStartSLOduration=3.069445177 podStartE2EDuration="22.310912265s" totalImagesPullingTime="19.241467088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:04:59.310367209 +0000 UTC m=+23.948226339" watchObservedRunningTime="2026-07-28 22:04:59.310912265 +0000 UTC m=+23.948771393" Jul 28 22:04:59.351733 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.351675 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9jwvn" podStartSLOduration=2.620028863 podStartE2EDuration="23.351648964s" totalImagesPullingTime="20.731620101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:04:59.351125211 +0000 UTC m=+23.988984417" watchObservedRunningTime="2026-07-28 22:04:59.351648964 +0000 UTC m=+23.989508094" Jul 28 22:04:59.771897 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.771737 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 22:04:59.852697 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.852563 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T22:04:59.771891093Z","UUID":"b8849326-6ea8-4c0e-ada3-56318a31f276","Handler":null,"Name":"","Endpoint":""} Jul 28 22:04:59.854198 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.854178 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 22:04:59.854322 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:04:59.854223 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 22:05:00.040525 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:00.040437 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:00.040839 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:00.040443 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:00.040839 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:00.040576 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:00.040839 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:00.040671 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:00.276277 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:00.276236 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" event={"ID":"4f7813ac-e730-4a92-8bc9-121af675b07d","Type":"ContainerStarted","Data":"263bf0ae677761cd9b485f713248adc305fbb9bf8353a62c52e0af98744f50b0"} Jul 28 22:05:00.278517 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:00.278465 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" event={"ID":"4cb85e1441cf636ff6c526ecb5a002c1","Type":"ContainerStarted","Data":"408c23b26e1051a64e7f19ca10e5081e326f6e54034134e42563d634e27265a0"} Jul 28 22:05:00.280240 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:00.280215 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tmrzl" event={"ID":"9f659954-188f-4265-a463-cfd4ede0baa7","Type":"ContainerStarted","Data":"1efc02fe2fb980ea28031befc622a08fbf2a9cfd8c3dd9a74793d65d2a60fa47"} Jul 28 22:05:00.293877 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:00.293789 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-225.ec2.internal" podStartSLOduration=23.293771655 podStartE2EDuration="23.293771655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:00.293715571 +0000 UTC m=+24.931574701" watchObservedRunningTime="2026-07-28 22:05:00.293771655 +0000 UTC m=+24.931630784" Jul 28 22:05:00.307888 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:00.307840 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tmrzl" podStartSLOduration=3.614134482 podStartE2EDuration="24.307823826s" totalImagesPullingTime="20.693689344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:00.307510358 +0000 UTC m=+24.945369486" watchObservedRunningTime="2026-07-28 22:05:00.307823826 +0000 UTC m=+24.945682953" Jul 28 22:05:01.040072 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:01.040046 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:01.040514 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:01.040175 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:01.286159 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:01.286075 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" event={"ID":"1232cacc-c457-432f-99fc-90fcd75903a5","Type":"ContainerStarted","Data":"a4c811d28a0d61b1e2ad91ff940e0451d7d82be21ccfefb524f414fe4d45fb70"} Jul 28 22:05:01.288289 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:01.288263 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" event={"ID":"4f7813ac-e730-4a92-8bc9-121af675b07d","Type":"ContainerStarted","Data":"bc4ec3e9963dbf5309552de78600b80506a2b7eacda96aefcec9a754688c37b1"} Jul 28 22:05:01.306997 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:01.306952 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgsfb" podStartSLOduration=2.276433983 podStartE2EDuration="25.306938372s" totalImagesPullingTime="23.030504389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:01.306689866 +0000 UTC m=+25.944548997" watchObservedRunningTime="2026-07-28 22:05:01.306938372 +0000 UTC m=+25.944797498" Jul 28 22:05:02.040234 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:02.040187 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:02.040440 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:02.040187 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:02.040440 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:02.040313 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:02.040440 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:02.040415 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:02.842490 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:02.842464 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-65mzx_c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57/dns-node-resolver/0.log" Jul 28 22:05:03.039534 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:03.039502 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:03.039695 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:03.039630 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:03.821127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:03.820947 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5zjn6_f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470/node-ca/0.log" Jul 28 22:05:04.039337 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.039309 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:04.039879 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:04.039446 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:04.039879 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.039488 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:04.039879 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:04.039566 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:04.250105 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.250071 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:05:04.250706 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.250687 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:05:04.298672 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.298581 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" event={"ID":"1232cacc-c457-432f-99fc-90fcd75903a5","Type":"ContainerStarted","Data":"43eb3089b4fccdd30a8bd513d35034725b461e4698a10bc5425dd2cec5667c0c"} Jul 28 22:05:04.298850 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.298830 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:05:04.300439 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.300415 2557 generic.go:356] "Generic (PLEG): container finished" podID="5137a1e9-ebea-49c3-9d66-9f836d3bb074" containerID="f5126f4fb88b949089d8266f7444293b06ccb137cb1955b010c9ad848f795d2f" exitCode=0 Jul 28 22:05:04.300565 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.300486 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" event={"ID":"5137a1e9-ebea-49c3-9d66-9f836d3bb074","Type":"ContainerDied","Data":"f5126f4fb88b949089d8266f7444293b06ccb137cb1955b010c9ad848f795d2f"} Jul 28 22:05:04.300677 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.300660 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:05:04.301632 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.301129 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9jwvn" Jul 28 22:05:04.315393 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.315371 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:05:04.334252 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:04.334207 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" podStartSLOduration=7.308069517 podStartE2EDuration="28.334192642s" totalImagesPullingTime="21.026123125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:04.333472713 +0000 UTC m=+28.971331842" watchObservedRunningTime="2026-07-28 22:05:04.334192642 +0000 UTC m=+28.972051792" Jul 28 22:05:05.040180 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.040148 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:05.040682 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:05.040251 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:05.302797 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.302718 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:05:05.302962 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.302802 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:05:05.317637 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.317610 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:05:05.777650 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.777622 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vg2d2"] Jul 28 22:05:05.777822 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.777760 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:05.777896 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:05.777877 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:05.779750 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.779721 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6fnc"] Jul 28 22:05:05.779912 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.779864 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:05.780045 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:05.779994 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:05.780479 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.780456 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-76hsd"] Jul 28 22:05:05.780578 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:05.780561 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:05.780676 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:05.780656 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:07.039545 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:07.039361 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:07.040031 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:07.039361 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:07.040031 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:07.039617 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:07.040031 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:07.039688 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:07.309054 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:07.309017 2557 generic.go:356] "Generic (PLEG): container finished" podID="5137a1e9-ebea-49c3-9d66-9f836d3bb074" containerID="564fe089781d39423e5eafe92aee7c6820ca5ce0b78a2a8164921fd09125bab2" exitCode=0 Jul 28 22:05:07.309199 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:07.309095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" event={"ID":"5137a1e9-ebea-49c3-9d66-9f836d3bb074","Type":"ContainerDied","Data":"564fe089781d39423e5eafe92aee7c6820ca5ce0b78a2a8164921fd09125bab2"} Jul 28 22:05:08.039714 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:08.039662 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:08.040191 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:08.039791 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:08.541118 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:08.541088 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:08.541245 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:08.541129 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:08.541245 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:08.541235 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:05:08.541312 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:08.541294 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret podName:e7b82635-0361-44b9-9b97-7f496d2be84c nodeName:}" failed. No retries permitted until 2026-07-28 22:05:40.5412812 +0000 UTC m=+65.179140309 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret") pod "global-pull-secret-syncer-76hsd" (UID: "e7b82635-0361-44b9-9b97-7f496d2be84c") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:05:08.541352 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:08.541235 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:05:08.541388 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:08.541361 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs podName:81accbfc-3d54-4286-87ff-3a6a34407c5d nodeName:}" failed. No retries permitted until 2026-07-28 22:05:40.541349133 +0000 UTC m=+65.179208258 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs") pod "network-metrics-daemon-vg2d2" (UID: "81accbfc-3d54-4286-87ff-3a6a34407c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:05:08.641680 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:08.641641 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:08.641804 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:08.641783 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:05:08.641804 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:08.641802 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:05:08.641885 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:08.641811 2557 projected.go:196] Error preparing data for projected volume kube-api-access-jvsjg for pod openshift-network-diagnostics/network-check-target-g6fnc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:05:08.641885 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:08.641860 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg podName:b0c0f55b-3d81-4bf6-8085-5e2a335e5799 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:40.641843932 +0000 UTC m=+65.279703039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvsjg" (UniqueName: "kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg") pod "network-check-target-g6fnc" (UID: "b0c0f55b-3d81-4bf6-8085-5e2a335e5799") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:05:09.040138 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:09.040101 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:09.040545 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:09.040102 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:09.040545 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:09.040203 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:09.040545 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:09.040283 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:09.315648 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:09.315566 2557 generic.go:356] "Generic (PLEG): container finished" podID="5137a1e9-ebea-49c3-9d66-9f836d3bb074" containerID="2f90469652f07b2497018a9a7e2b6d6e93b690c2df9b0d5768dd930f089fa0b4" exitCode=0 Jul 28 22:05:09.315648 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:09.315605 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" event={"ID":"5137a1e9-ebea-49c3-9d66-9f836d3bb074","Type":"ContainerDied","Data":"2f90469652f07b2497018a9a7e2b6d6e93b690c2df9b0d5768dd930f089fa0b4"} Jul 28 22:05:10.039608 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:10.039578 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:10.039793 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:10.039697 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:11.039994 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:11.039958 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:11.040433 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:11.039958 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:11.040433 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:11.040092 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:11.040433 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:11.040191 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:12.039597 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:12.039550 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:12.039780 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:12.039702 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:13.039944 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:13.039894 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:13.039944 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:13.039915 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:13.040400 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:13.040035 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:13.040400 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:13.040169 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:13.327450 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:13.327419 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" event={"ID":"5137a1e9-ebea-49c3-9d66-9f836d3bb074","Type":"ContainerStarted","Data":"9ff6b936031ede98ec8fa55980dea1148dc9c1253764a100170bdff3c4642c38"} Jul 28 22:05:14.039337 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:14.039304 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:14.039539 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:14.039430 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:14.331969 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:14.331869 2557 generic.go:356] "Generic (PLEG): container finished" podID="5137a1e9-ebea-49c3-9d66-9f836d3bb074" containerID="9ff6b936031ede98ec8fa55980dea1148dc9c1253764a100170bdff3c4642c38" exitCode=0 Jul 28 22:05:14.331969 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:14.331911 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" event={"ID":"5137a1e9-ebea-49c3-9d66-9f836d3bb074","Type":"ContainerDied","Data":"9ff6b936031ede98ec8fa55980dea1148dc9c1253764a100170bdff3c4642c38"} Jul 28 22:05:15.039903 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:15.039707 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:15.040117 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:15.039707 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:15.040117 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:15.040023 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:15.040117 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:15.040067 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:15.338127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:15.338046 2557 generic.go:356] "Generic (PLEG): container finished" podID="5137a1e9-ebea-49c3-9d66-9f836d3bb074" containerID="0a86767665f833a251ca5bdb2e5c48e56ebfd2672fab7924a082dcc18c6600af" exitCode=0 Jul 28 22:05:15.338127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:15.338102 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" event={"ID":"5137a1e9-ebea-49c3-9d66-9f836d3bb074","Type":"ContainerDied","Data":"0a86767665f833a251ca5bdb2e5c48e56ebfd2672fab7924a082dcc18c6600af"} Jul 28 22:05:16.039998 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:16.039972 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:16.040161 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:16.040057 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:16.342853 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:16.342769 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" event={"ID":"5137a1e9-ebea-49c3-9d66-9f836d3bb074","Type":"ContainerStarted","Data":"e9c3870e792ac03124d5b1348e7f3ffab7e788700cef723f4932128bd53710bf"} Jul 28 22:05:16.368345 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:16.368278 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b5qcq" podStartSLOduration=8.29074 podStartE2EDuration="40.36825213s" totalImagesPullingTime="32.07751213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:16.365993812 +0000 UTC m=+41.003852941" watchObservedRunningTime="2026-07-28 22:05:16.36825213 +0000 UTC m=+41.006111250" Jul 28 22:05:17.039990 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:17.039959 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:17.039990 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:17.039990 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:17.040196 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:17.040063 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:17.040196 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:17.040165 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:18.039547 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:18.039513 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:18.040022 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:18.039652 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:19.040310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:19.040265 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:19.040709 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:19.040280 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:19.040709 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:19.040395 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:19.040709 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:19.040435 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:20.040193 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:20.040157 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:20.040368 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:20.040323 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:21.039621 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:21.039589 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:21.039869 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:21.039587 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:21.039869 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:21.039693 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:21.039869 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:21.039751 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:22.039633 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:22.039599 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:22.040038 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:22.039721 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:23.039943 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:23.039899 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:23.040319 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:23.040015 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:23.040319 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:23.040071 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:23.040319 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:23.040154 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:24.039583 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:24.039546 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:24.039751 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:24.039670 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vg2d2" podUID="81accbfc-3d54-4286-87ff-3a6a34407c5d" Jul 28 22:05:25.040025 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.039988 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:25.040401 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.039988 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:25.040401 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:25.040107 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6fnc" podUID="b0c0f55b-3d81-4bf6-8085-5e2a335e5799" Jul 28 22:05:25.040401 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:25.040157 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76hsd" podUID="e7b82635-0361-44b9-9b97-7f496d2be84c" Jul 28 22:05:25.672824 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.672797 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-225.ec2.internal" event="NodeReady" Jul 28 22:05:25.672997 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.672952 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 22:05:25.722694 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.722663 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k7xl5"] Jul 28 22:05:25.736046 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.736018 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zztbv"] Jul 28 22:05:25.736180 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.736136 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.738806 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.738782 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 22:05:25.738969 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.738946 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 22:05:25.739056 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.739007 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8xbs4\"" Jul 28 22:05:25.748727 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.748707 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k7xl5"] Jul 28 22:05:25.748842 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.748751 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zztbv"] Jul 28 22:05:25.748842 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.748820 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zztbv" Jul 28 22:05:25.753353 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.753311 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 22:05:25.753490 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.753461 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 22:05:25.753490 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.753479 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zn58b\"" Jul 28 22:05:25.753613 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.753510 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 22:05:25.824217 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.824140 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7cfns"] Jul 28 22:05:25.841353 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.841326 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7cfns"] Jul 28 22:05:25.841488 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.841475 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:25.844035 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.844010 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 22:05:25.844168 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.844082 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8gj46\"" Jul 28 22:05:25.844237 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.844223 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 22:05:25.844295 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.844223 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 22:05:25.844347 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.844302 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 22:05:25.869677 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.869652 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b009deee-53a6-4016-ba9f-a05750866a41-metrics-tls\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.869813 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.869684 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b009deee-53a6-4016-ba9f-a05750866a41-tmp-dir\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.869813 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.869709 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckl7l\" (UniqueName: \"kubernetes.io/projected/482beaea-592d-475b-9e38-c1be4dd15959-kube-api-access-ckl7l\") pod \"ingress-canary-zztbv\" (UID: \"482beaea-592d-475b-9e38-c1be4dd15959\") " pod="openshift-ingress-canary/ingress-canary-zztbv" Jul 28 22:05:25.869813 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.869765 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b009deee-53a6-4016-ba9f-a05750866a41-config-volume\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.869813 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.869793 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482beaea-592d-475b-9e38-c1be4dd15959-cert\") pod \"ingress-canary-zztbv\" (UID: \"482beaea-592d-475b-9e38-c1be4dd15959\") " pod="openshift-ingress-canary/ingress-canary-zztbv" Jul 28 22:05:25.869960 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.869826 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9glvz\" (UniqueName: \"kubernetes.io/projected/b009deee-53a6-4016-ba9f-a05750866a41-kube-api-access-9glvz\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.971081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b009deee-53a6-4016-ba9f-a05750866a41-config-volume\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.971081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971078 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482beaea-592d-475b-9e38-c1be4dd15959-cert\") pod \"ingress-canary-zztbv\" (UID: \"482beaea-592d-475b-9e38-c1be4dd15959\") " pod="openshift-ingress-canary/ingress-canary-zztbv" Jul 28 22:05:25.971308 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971115 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7faf624-53f7-4110-bf99-9dd1202d493e-crio-socket\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:25.971308 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971155 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7faf624-53f7-4110-bf99-9dd1202d493e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:25.971308 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971176 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9glvz\" (UniqueName: \"kubernetes.io/projected/b009deee-53a6-4016-ba9f-a05750866a41-kube-api-access-9glvz\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.971308 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b009deee-53a6-4016-ba9f-a05750866a41-metrics-tls\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.971508 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971450 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7faf624-53f7-4110-bf99-9dd1202d493e-data-volume\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:25.971508 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971484 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b009deee-53a6-4016-ba9f-a05750866a41-tmp-dir\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.971610 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971505 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlkmm\" (UniqueName: \"kubernetes.io/projected/b7faf624-53f7-4110-bf99-9dd1202d493e-kube-api-access-rlkmm\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:25.971666 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971638 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckl7l\" (UniqueName: \"kubernetes.io/projected/482beaea-592d-475b-9e38-c1be4dd15959-kube-api-access-ckl7l\") pod \"ingress-canary-zztbv\" (UID: \"482beaea-592d-475b-9e38-c1be4dd15959\") " pod="openshift-ingress-canary/ingress-canary-zztbv" Jul 28 22:05:25.971720 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971670 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7faf624-53f7-4110-bf99-9dd1202d493e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:25.971773 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971716 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b009deee-53a6-4016-ba9f-a05750866a41-config-volume\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.971773 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.971752 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b009deee-53a6-4016-ba9f-a05750866a41-tmp-dir\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.975351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.975326 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482beaea-592d-475b-9e38-c1be4dd15959-cert\") pod \"ingress-canary-zztbv\" (UID: \"482beaea-592d-475b-9e38-c1be4dd15959\") " pod="openshift-ingress-canary/ingress-canary-zztbv" Jul 28 22:05:25.975473 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.975329 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b009deee-53a6-4016-ba9f-a05750866a41-metrics-tls\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:25.979160 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.979138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckl7l\" (UniqueName: \"kubernetes.io/projected/482beaea-592d-475b-9e38-c1be4dd15959-kube-api-access-ckl7l\") pod \"ingress-canary-zztbv\" (UID: \"482beaea-592d-475b-9e38-c1be4dd15959\") " pod="openshift-ingress-canary/ingress-canary-zztbv" Jul 28 22:05:25.979296 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:25.979204 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9glvz\" (UniqueName: \"kubernetes.io/projected/b009deee-53a6-4016-ba9f-a05750866a41-kube-api-access-9glvz\") pod \"dns-default-k7xl5\" (UID: \"b009deee-53a6-4016-ba9f-a05750866a41\") " pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:26.045222 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.045191 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:26.045720 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.045544 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:26.047607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.047587 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sm6xq\"" Jul 28 22:05:26.047710 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.047593 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 22:05:26.057222 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.057206 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zztbv" Jul 28 22:05:26.072803 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.072783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7faf624-53f7-4110-bf99-9dd1202d493e-crio-socket\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.072884 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.072811 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7faf624-53f7-4110-bf99-9dd1202d493e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.072985 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.072969 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7faf624-53f7-4110-bf99-9dd1202d493e-data-volume\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.073048 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.073013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlkmm\" (UniqueName: \"kubernetes.io/projected/b7faf624-53f7-4110-bf99-9dd1202d493e-kube-api-access-rlkmm\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.073048 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.073023 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7faf624-53f7-4110-bf99-9dd1202d493e-crio-socket\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.073149 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.073060 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7faf624-53f7-4110-bf99-9dd1202d493e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.073329 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.073307 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7faf624-53f7-4110-bf99-9dd1202d493e-data-volume\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.073395 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.073359 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7faf624-53f7-4110-bf99-9dd1202d493e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.075317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.075297 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7faf624-53f7-4110-bf99-9dd1202d493e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.084594 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.084567 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlkmm\" (UniqueName: \"kubernetes.io/projected/b7faf624-53f7-4110-bf99-9dd1202d493e-kube-api-access-rlkmm\") pod \"insights-runtime-extractor-7cfns\" (UID: \"b7faf624-53f7-4110-bf99-9dd1202d493e\") " pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.151068 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.150880 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7cfns" Jul 28 22:05:26.257322 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.257292 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k7xl5"] Jul 28 22:05:26.260865 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:26.260836 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb009deee_53a6_4016_ba9f_a05750866a41.slice/crio-20da01e6e3e2c1c95020f187e98cd52ef3ca22774d51de49abca7972f4045108 WatchSource:0}: Error finding container 20da01e6e3e2c1c95020f187e98cd52ef3ca22774d51de49abca7972f4045108: Status 404 returned error can't find the container with id 20da01e6e3e2c1c95020f187e98cd52ef3ca22774d51de49abca7972f4045108 Jul 28 22:05:26.270743 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.270718 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zztbv"] Jul 28 22:05:26.275150 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:26.275114 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482beaea_592d_475b_9e38_c1be4dd15959.slice/crio-774933078c56f1d6f398a40723d2d5f6af3b1715660ad3c3b9dad97cdc19558c WatchSource:0}: Error finding container 774933078c56f1d6f398a40723d2d5f6af3b1715660ad3c3b9dad97cdc19558c: Status 404 returned error can't find the container with id 774933078c56f1d6f398a40723d2d5f6af3b1715660ad3c3b9dad97cdc19558c Jul 28 22:05:26.319583 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.319553 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7cfns"] Jul 28 22:05:26.322632 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:26.322598 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7faf624_53f7_4110_bf99_9dd1202d493e.slice/crio-45803421a0609f83594399ce95c8b5dd7fde8086caa9d3e38a08996c5040b4a6 WatchSource:0}: Error finding container 45803421a0609f83594399ce95c8b5dd7fde8086caa9d3e38a08996c5040b4a6: Status 404 returned error can't find the container with id 45803421a0609f83594399ce95c8b5dd7fde8086caa9d3e38a08996c5040b4a6 Jul 28 22:05:26.366688 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.366656 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cfns" event={"ID":"b7faf624-53f7-4110-bf99-9dd1202d493e","Type":"ContainerStarted","Data":"45803421a0609f83594399ce95c8b5dd7fde8086caa9d3e38a08996c5040b4a6"} Jul 28 22:05:26.367720 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.367694 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zztbv" event={"ID":"482beaea-592d-475b-9e38-c1be4dd15959","Type":"ContainerStarted","Data":"774933078c56f1d6f398a40723d2d5f6af3b1715660ad3c3b9dad97cdc19558c"} Jul 28 22:05:26.368638 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:26.368617 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k7xl5" event={"ID":"b009deee-53a6-4016-ba9f-a05750866a41","Type":"ContainerStarted","Data":"20da01e6e3e2c1c95020f187e98cd52ef3ca22774d51de49abca7972f4045108"} Jul 28 22:05:27.039437 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:27.039404 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:27.039609 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:27.039561 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:27.054173 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:27.054154 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 22:05:27.054448 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:27.054221 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnqzv\"" Jul 28 22:05:27.054448 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:27.054352 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 22:05:27.055236 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:27.055223 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 22:05:30.438497 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.438464 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-776d948f57-5692h"] Jul 28 22:05:30.471440 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.471404 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776d948f57-5692h"] Jul 28 22:05:30.471594 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.471502 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.474552 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.474529 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 22:05:30.474842 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.474815 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 22:05:30.474842 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.474840 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 22:05:30.475395 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.475375 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 22:05:30.475481 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.475406 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 22:05:30.475844 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.475826 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2k6rq\"" Jul 28 22:05:30.475956 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.475903 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 22:05:30.475956 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.475918 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 22:05:30.605114 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.605071 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-config\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.605278 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.605147 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-oauth-serving-cert\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.605278 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.605253 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-service-ca\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.605367 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.605284 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-serving-cert\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.605367 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.605324 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-oauth-config\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.605367 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.605343 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p77q\" (UniqueName: \"kubernetes.io/projected/16902eed-a0c9-4180-8b79-3cb0795cd6d9-kube-api-access-4p77q\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.706019 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.705910 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-config\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.706019 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.705987 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-oauth-serving-cert\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.706207 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.706028 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-service-ca\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.706207 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.706051 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-serving-cert\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.706207 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.706085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-oauth-config\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.706207 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.706110 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p77q\" (UniqueName: \"kubernetes.io/projected/16902eed-a0c9-4180-8b79-3cb0795cd6d9-kube-api-access-4p77q\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.707263 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.707197 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-config\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.707263 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.707198 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-oauth-serving-cert\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.707447 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.707265 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-service-ca\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.709002 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.708982 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-oauth-config\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.709002 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.708988 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-serving-cert\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.715417 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.715370 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p77q\" (UniqueName: \"kubernetes.io/projected/16902eed-a0c9-4180-8b79-3cb0795cd6d9-kube-api-access-4p77q\") pod \"console-776d948f57-5692h\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:30.782325 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:30.782288 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:32.623585 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.623545 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bf8854547-qcp2f"] Jul 28 22:05:32.626077 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.626056 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.635425 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.635370 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 22:05:32.636690 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.636657 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf8854547-qcp2f"] Jul 28 22:05:32.823350 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.823322 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhvm\" (UniqueName: \"kubernetes.io/projected/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-kube-api-access-twhvm\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.823467 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.823391 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-service-ca\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.823467 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.823418 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-oauth-serving-cert\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.823583 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.823477 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-oauth-config\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.823583 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.823502 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-config\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.823583 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.823539 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-serving-cert\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.823583 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.823562 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-trusted-ca-bundle\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.886215 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.886179 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776d948f57-5692h"] Jul 28 22:05:32.896631 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:32.896587 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16902eed_a0c9_4180_8b79_3cb0795cd6d9.slice/crio-1185c22aef62a9e191980f1a601e168d0fdc24a4a343062a83fa942bcf2205eb WatchSource:0}: Error finding container 1185c22aef62a9e191980f1a601e168d0fdc24a4a343062a83fa942bcf2205eb: Status 404 returned error can't find the container with id 1185c22aef62a9e191980f1a601e168d0fdc24a4a343062a83fa942bcf2205eb Jul 28 22:05:32.924409 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.924375 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhvm\" (UniqueName: \"kubernetes.io/projected/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-kube-api-access-twhvm\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.924640 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.924435 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-service-ca\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.924640 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.924457 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-oauth-serving-cert\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.924640 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.924507 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-oauth-config\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.924804 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.924677 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-config\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.924804 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.924744 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-serving-cert\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.924804 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.924771 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-trusted-ca-bundle\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.925465 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.925245 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-service-ca\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.925465 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.925346 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-oauth-serving-cert\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.926136 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.926089 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-config\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.926247 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.926138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-trusted-ca-bundle\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.927336 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.927293 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-oauth-config\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.927950 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.927918 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-serving-cert\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:32.942773 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:32.942749 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhvm\" (UniqueName: \"kubernetes.io/projected/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-kube-api-access-twhvm\") pod \"console-5bf8854547-qcp2f\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:33.208812 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.208724 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx"] Jul 28 22:05:33.210614 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.210594 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" Jul 28 22:05:33.212909 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.212888 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 28 22:05:33.213033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.213015 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bbvzk\"" Jul 28 22:05:33.219587 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.219566 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx"] Jul 28 22:05:33.227122 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.227099 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/538621a7-527c-4606-88e0-95bd3c4c4042-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-pf6mx\" (UID: \"538621a7-527c-4606-88e0-95bd3c4c4042\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" Jul 28 22:05:33.236439 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.236419 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:33.327835 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.327776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/538621a7-527c-4606-88e0-95bd3c4c4042-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-pf6mx\" (UID: \"538621a7-527c-4606-88e0-95bd3c4c4042\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" Jul 28 22:05:33.328010 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:33.327985 2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 28 22:05:33.328083 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:33.328071 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538621a7-527c-4606-88e0-95bd3c4c4042-tls-certificates podName:538621a7-527c-4606-88e0-95bd3c4c4042 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:33.828048925 +0000 UTC m=+58.465908038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/538621a7-527c-4606-88e0-95bd3c4c4042-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" (UID: "538621a7-527c-4606-88e0-95bd3c4c4042") : secret "prometheus-operator-admission-webhook-tls" not found Jul 28 22:05:33.360687 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.360651 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf8854547-qcp2f"] Jul 28 22:05:33.363394 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:33.363365 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacdc7b8_0c5e_44a2_b600_0e9b903ff402.slice/crio-55887fdd8eea4980e98d79c82931b87b107d51d014edaaa3c89bc04980cd8b79 WatchSource:0}: Error finding container 55887fdd8eea4980e98d79c82931b87b107d51d014edaaa3c89bc04980cd8b79: Status 404 returned error can't find the container with id 55887fdd8eea4980e98d79c82931b87b107d51d014edaaa3c89bc04980cd8b79 Jul 28 22:05:33.389906 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.389875 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zztbv" event={"ID":"482beaea-592d-475b-9e38-c1be4dd15959","Type":"ContainerStarted","Data":"610d94f32386233ddf93d88c09fa4b825c4d31010347be60bce74f076395f793"} Jul 28 22:05:33.391290 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.391266 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cfns" event={"ID":"b7faf624-53f7-4110-bf99-9dd1202d493e","Type":"ContainerStarted","Data":"512647073bcca84d40994416d4a02be9ee0f4a82b23ea78b52d14107c6f38063"} Jul 28 22:05:33.392433 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.392408 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776d948f57-5692h" event={"ID":"16902eed-a0c9-4180-8b79-3cb0795cd6d9","Type":"ContainerStarted","Data":"1185c22aef62a9e191980f1a601e168d0fdc24a4a343062a83fa942bcf2205eb"} Jul 28 22:05:33.394083 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.394062 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k7xl5" event={"ID":"b009deee-53a6-4016-ba9f-a05750866a41","Type":"ContainerStarted","Data":"1d823806c39055cc8ec256c5d033493b3de5f944fc496a51ffc461c3b36b9088"} Jul 28 22:05:33.394220 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.394109 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:33.394220 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.394133 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k7xl5" event={"ID":"b009deee-53a6-4016-ba9f-a05750866a41","Type":"ContainerStarted","Data":"9e37daad51d09d52537cb3d82b8c94710c07b26d4c2163367ec444330ac54989"} Jul 28 22:05:33.396568 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.396540 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf8854547-qcp2f" event={"ID":"aacdc7b8-0c5e-44a2-b600-0e9b903ff402","Type":"ContainerStarted","Data":"55887fdd8eea4980e98d79c82931b87b107d51d014edaaa3c89bc04980cd8b79"} Jul 28 22:05:33.409422 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.409378 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zztbv" podStartSLOduration=1.946312032 podStartE2EDuration="8.409363895s" totalImagesPullingTime="6.463051863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:33.409074926 +0000 UTC m=+58.046934055" watchObservedRunningTime="2026-07-28 22:05:33.409363895 +0000 UTC m=+58.047223002" Jul 28 22:05:33.427483 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.427237 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k7xl5" podStartSLOduration=1.950823097 podStartE2EDuration="8.427216355s" totalImagesPullingTime="6.476393258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:33.426670688 +0000 UTC m=+58.064529846" watchObservedRunningTime="2026-07-28 22:05:33.427216355 +0000 UTC m=+58.065075482" Jul 28 22:05:33.832667 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.832085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/538621a7-527c-4606-88e0-95bd3c4c4042-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-pf6mx\" (UID: \"538621a7-527c-4606-88e0-95bd3c4c4042\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" Jul 28 22:05:33.836628 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:33.836573 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/538621a7-527c-4606-88e0-95bd3c4c4042-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-pf6mx\" (UID: \"538621a7-527c-4606-88e0-95bd3c4c4042\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" Jul 28 22:05:34.119805 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:34.119631 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" Jul 28 22:05:34.294428 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:34.294125 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx"] Jul 28 22:05:34.409210 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:34.409125 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" event={"ID":"538621a7-527c-4606-88e0-95bd3c4c4042","Type":"ContainerStarted","Data":"08eb09cf7d4f683ae1f523f79562298b14bdf26411c7e545ccd45930af6ce502"} Jul 28 22:05:37.327414 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:37.327383 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtzj4" Jul 28 22:05:39.425317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:39.425231 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cfns" event={"ID":"b7faf624-53f7-4110-bf99-9dd1202d493e","Type":"ContainerStarted","Data":"ddcc6491884e2fd5d2b2048cda15bac35e602c98729c0d449a0d551a6db26289"} Jul 28 22:05:39.426576 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:39.426556 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776d948f57-5692h" event={"ID":"16902eed-a0c9-4180-8b79-3cb0795cd6d9","Type":"ContainerStarted","Data":"20d60ff2c724abebe6a8210a89628b0de2a5cc7bac006f6428b2721e3463feed"} Jul 28 22:05:39.427910 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:39.427888 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf8854547-qcp2f" event={"ID":"aacdc7b8-0c5e-44a2-b600-0e9b903ff402","Type":"ContainerStarted","Data":"fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783"} Jul 28 22:05:39.429227 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:39.429208 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" event={"ID":"538621a7-527c-4606-88e0-95bd3c4c4042","Type":"ContainerStarted","Data":"c681920dec7a192787a50657cc614eb26fdffe65622a7efc733fcd9780778f48"} Jul 28 22:05:39.429380 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:39.429361 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" Jul 28 22:05:39.433895 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:39.433876 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" Jul 28 22:05:39.450624 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:39.450583 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-776d948f57-5692h" podStartSLOduration=3.294426013 podStartE2EDuration="9.450569283s" totalImagesPullingTime="6.15614327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:39.450053807 +0000 UTC m=+64.087912934" watchObservedRunningTime="2026-07-28 22:05:39.450569283 +0000 UTC m=+64.088428402" Jul 28 22:05:39.468677 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:39.468632 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-pf6mx" podStartSLOduration=1.718916154 podStartE2EDuration="6.468617272s" totalImagesPullingTime="4.749701118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:39.468317372 +0000 UTC m=+64.106176513" watchObservedRunningTime="2026-07-28 22:05:39.468617272 +0000 UTC m=+64.106476401" Jul 28 22:05:39.489453 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:39.489399 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bf8854547-qcp2f" podStartSLOduration=1.770989897 podStartE2EDuration="7.489381317s" totalImagesPullingTime="5.71839142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:39.488628538 +0000 UTC m=+64.126487666" watchObservedRunningTime="2026-07-28 22:05:39.489381317 +0000 UTC m=+64.127240440" Jul 28 22:05:40.280711 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.280678 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6xc8m"] Jul 28 22:05:40.283009 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.282989 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.285609 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.285586 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 22:05:40.285825 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.285805 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 22:05:40.286696 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.286622 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rgw24\"" Jul 28 22:05:40.286696 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.286676 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 22:05:40.286887 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.286676 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 22:05:40.287031 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.287011 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 22:05:40.295217 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.295194 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6xc8m"] Jul 28 22:05:40.380143 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.380111 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c10b52f5-59ed-4e92-a868-f3844534e1e0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.380143 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.380148 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdqcp\" (UniqueName: \"kubernetes.io/projected/c10b52f5-59ed-4e92-a868-f3844534e1e0-kube-api-access-hdqcp\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.380434 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.380268 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10b52f5-59ed-4e92-a868-f3844534e1e0-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.380434 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.380300 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c10b52f5-59ed-4e92-a868-f3844534e1e0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.480762 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.480730 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c10b52f5-59ed-4e92-a868-f3844534e1e0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.481218 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.480767 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdqcp\" (UniqueName: \"kubernetes.io/projected/c10b52f5-59ed-4e92-a868-f3844534e1e0-kube-api-access-hdqcp\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.481350 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.481323 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10b52f5-59ed-4e92-a868-f3844534e1e0-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.481842 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.481815 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c10b52f5-59ed-4e92-a868-f3844534e1e0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.482102 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.482075 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10b52f5-59ed-4e92-a868-f3844534e1e0-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.482328 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:40.482311 2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 28 22:05:40.482392 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:40.482384 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10b52f5-59ed-4e92-a868-f3844534e1e0-prometheus-operator-tls podName:c10b52f5-59ed-4e92-a868-f3844534e1e0 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:40.982365049 +0000 UTC m=+65.620224158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c10b52f5-59ed-4e92-a868-f3844534e1e0-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-6xc8m" (UID: "c10b52f5-59ed-4e92-a868-f3844534e1e0") : secret "prometheus-operator-tls" not found Jul 28 22:05:40.483630 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.483548 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c10b52f5-59ed-4e92-a868-f3844534e1e0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.498997 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.498969 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdqcp\" (UniqueName: \"kubernetes.io/projected/c10b52f5-59ed-4e92-a868-f3844534e1e0-kube-api-access-hdqcp\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.583274 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.583185 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:40.583274 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.583266 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:40.586193 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.586163 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 22:05:40.586344 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.586218 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 22:05:40.595918 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.595895 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81accbfc-3d54-4286-87ff-3a6a34407c5d-metrics-certs\") pod \"network-metrics-daemon-vg2d2\" (UID: \"81accbfc-3d54-4286-87ff-3a6a34407c5d\") " pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:40.596072 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.596011 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b82635-0361-44b9-9b97-7f496d2be84c-original-pull-secret\") pod \"global-pull-secret-syncer-76hsd\" (UID: \"e7b82635-0361-44b9-9b97-7f496d2be84c\") " pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:40.684037 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.684004 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:40.687171 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.687151 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 22:05:40.697983 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.697959 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 22:05:40.707983 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.707958 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsjg\" (UniqueName: \"kubernetes.io/projected/b0c0f55b-3d81-4bf6-8085-5e2a335e5799-kube-api-access-jvsjg\") pod \"network-check-target-g6fnc\" (UID: \"b0c0f55b-3d81-4bf6-8085-5e2a335e5799\") " pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:40.758583 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.758552 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sm6xq\"" Jul 28 22:05:40.765162 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.765139 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vg2d2" Jul 28 22:05:40.783099 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.783072 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:40.783212 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.783141 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:40.789084 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.789047 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:40.848938 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.848846 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76hsd" Jul 28 22:05:40.857469 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.857303 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnqzv\"" Jul 28 22:05:40.864969 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.864868 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:40.912686 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.912571 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vg2d2"] Jul 28 22:05:40.931187 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:40.931143 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81accbfc_3d54_4286_87ff_3a6a34407c5d.slice/crio-0afd3509c91b91f35ef35297e2b68aef332be0813be44bdb23e518c231b11576 WatchSource:0}: Error finding container 0afd3509c91b91f35ef35297e2b68aef332be0813be44bdb23e518c231b11576: Status 404 returned error can't find the container with id 0afd3509c91b91f35ef35297e2b68aef332be0813be44bdb23e518c231b11576 Jul 28 22:05:40.986921 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.986884 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c10b52f5-59ed-4e92-a868-f3844534e1e0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:40.990003 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:40.989973 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c10b52f5-59ed-4e92-a868-f3844534e1e0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6xc8m\" (UID: \"c10b52f5-59ed-4e92-a868-f3844534e1e0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:41.007850 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:41.007799 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-76hsd"] Jul 28 22:05:41.013350 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:41.013319 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b82635_0361_44b9_9b97_7f496d2be84c.slice/crio-4697e63b0e434e0fe592f085fa3eb74d19632cfb614ef32afffe89de4c7f824e WatchSource:0}: Error finding container 4697e63b0e434e0fe592f085fa3eb74d19632cfb614ef32afffe89de4c7f824e: Status 404 returned error can't find the container with id 4697e63b0e434e0fe592f085fa3eb74d19632cfb614ef32afffe89de4c7f824e Jul 28 22:05:41.034727 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:41.034697 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6fnc"] Jul 28 22:05:41.037981 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:41.037956 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c0f55b_3d81_4bf6_8085_5e2a335e5799.slice/crio-1380fdfb26238272bb57a474b8b55f8fcb60582c4bf7f6927e495a1a3742fda0 WatchSource:0}: Error finding container 1380fdfb26238272bb57a474b8b55f8fcb60582c4bf7f6927e495a1a3742fda0: Status 404 returned error can't find the container with id 1380fdfb26238272bb57a474b8b55f8fcb60582c4bf7f6927e495a1a3742fda0 Jul 28 22:05:41.194370 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:41.194287 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" Jul 28 22:05:41.327354 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:41.327326 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6xc8m"] Jul 28 22:05:41.437846 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:41.437810 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6fnc" event={"ID":"b0c0f55b-3d81-4bf6-8085-5e2a335e5799","Type":"ContainerStarted","Data":"1380fdfb26238272bb57a474b8b55f8fcb60582c4bf7f6927e495a1a3742fda0"} Jul 28 22:05:41.439089 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:41.439060 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-76hsd" event={"ID":"e7b82635-0361-44b9-9b97-7f496d2be84c","Type":"ContainerStarted","Data":"4697e63b0e434e0fe592f085fa3eb74d19632cfb614ef32afffe89de4c7f824e"} Jul 28 22:05:41.440494 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:41.440468 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vg2d2" event={"ID":"81accbfc-3d54-4286-87ff-3a6a34407c5d","Type":"ContainerStarted","Data":"0afd3509c91b91f35ef35297e2b68aef332be0813be44bdb23e518c231b11576"} Jul 28 22:05:41.444975 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:41.444902 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-776d948f57-5692h" Jul 28 22:05:42.903311 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:42.903274 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10b52f5_59ed_4e92_a868_f3844534e1e0.slice/crio-0203dcd47f07692503103ae7e51c96ad3e20a30ee805229e03a17248512c2ae8 WatchSource:0}: Error finding container 0203dcd47f07692503103ae7e51c96ad3e20a30ee805229e03a17248512c2ae8: Status 404 returned error can't find the container with id 0203dcd47f07692503103ae7e51c96ad3e20a30ee805229e03a17248512c2ae8 Jul 28 22:05:43.237853 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:43.237169 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:43.238079 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:43.237958 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:43.244783 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:43.244568 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:43.411378 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:43.411346 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k7xl5" Jul 28 22:05:43.454209 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:43.454175 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cfns" event={"ID":"b7faf624-53f7-4110-bf99-9dd1202d493e","Type":"ContainerStarted","Data":"78535db4c1d2f600d0b6f70a9643b621ad7a3541d05482c371b5dcc125119f75"} Jul 28 22:05:43.457495 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:43.457412 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" event={"ID":"c10b52f5-59ed-4e92-a868-f3844534e1e0","Type":"ContainerStarted","Data":"0203dcd47f07692503103ae7e51c96ad3e20a30ee805229e03a17248512c2ae8"} Jul 28 22:05:43.462591 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:43.462556 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:05:43.473558 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:43.472720 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7cfns" podStartSLOduration=1.9836036 podStartE2EDuration="18.472677662s" totalImagesPullingTime="16.489074062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:43.4714101 +0000 UTC m=+68.109269232" watchObservedRunningTime="2026-07-28 22:05:43.472677662 +0000 UTC m=+68.110536792" Jul 28 22:05:43.519355 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:43.519089 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776d948f57-5692h"] Jul 28 22:05:44.463225 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:44.463184 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vg2d2" event={"ID":"81accbfc-3d54-4286-87ff-3a6a34407c5d","Type":"ContainerStarted","Data":"30868dd7bd75255840839deda554697d3b50ff042079bfb72f7b047d853e6f76"} Jul 28 22:05:44.463698 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:44.463253 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vg2d2" event={"ID":"81accbfc-3d54-4286-87ff-3a6a34407c5d","Type":"ContainerStarted","Data":"4c874a43e18a9da9854a5e9ed0ef69ad760874dfcf789f8435ddc35789ba4952"} Jul 28 22:05:44.482043 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:44.481981 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vg2d2" podStartSLOduration=65.675503697 podStartE2EDuration="1m8.481961782s" totalImagesPullingTime="2.806458085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:44.480725257 +0000 UTC m=+69.118584387" watchObservedRunningTime="2026-07-28 22:05:44.481961782 +0000 UTC m=+69.119820908" Jul 28 22:05:48.482775 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:48.482733 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6fnc" event={"ID":"b0c0f55b-3d81-4bf6-8085-5e2a335e5799","Type":"ContainerStarted","Data":"e81894e7a34d89d2b25208ec8609c4d1f0e4ae3414328c8351574a4a954407f9"} Jul 28 22:05:48.483269 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:48.482890 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:05:48.484216 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:48.484186 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-76hsd" event={"ID":"e7b82635-0361-44b9-9b97-7f496d2be84c","Type":"ContainerStarted","Data":"b67cd5fb70e5a2997507d6d43c9b3bc89efb0b51be156d72f32d73f738fccc88"} Jul 28 22:05:48.485993 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:48.485972 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" event={"ID":"c10b52f5-59ed-4e92-a868-f3844534e1e0","Type":"ContainerStarted","Data":"108bf5a2f84a36d33c88a3efeee07e686e34680cec46b244ae37e9c3240e81fa"} Jul 28 22:05:48.486075 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:48.486008 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" event={"ID":"c10b52f5-59ed-4e92-a868-f3844534e1e0","Type":"ContainerStarted","Data":"ae02ea56a49ea6426806ae27f50bb080ba9f1ed17d9a0c89a5528e05f5e5e750"} Jul 28 22:05:48.501298 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:48.501249 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g6fnc" podStartSLOduration=66.006228238 podStartE2EDuration="1m12.501236657s" totalImagesPullingTime="6.495008419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:48.500366997 +0000 UTC m=+73.138226126" watchObservedRunningTime="2026-07-28 22:05:48.501236657 +0000 UTC m=+73.139095784" Jul 28 22:05:48.518711 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:48.518663 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-76hsd" podStartSLOduration=64.981989091 podStartE2EDuration="1m11.518648616s" totalImagesPullingTime="6.536659525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:48.517848895 +0000 UTC m=+73.155708020" watchObservedRunningTime="2026-07-28 22:05:48.518648616 +0000 UTC m=+73.156507744" Jul 28 22:05:48.534446 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:48.534401 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xc8m" podStartSLOduration=3.903094367 podStartE2EDuration="8.534385776s" totalImagesPullingTime="4.631291409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:48.533910246 +0000 UTC m=+73.171769374" watchObservedRunningTime="2026-07-28 22:05:48.534385776 +0000 UTC m=+73.172244904" Jul 28 22:05:50.610276 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.610248 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db"] Jul 28 22:05:50.612762 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.612746 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.615188 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.615167 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 22:05:50.615423 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.615407 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 22:05:50.615512 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.615422 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2gmz6\"" Jul 28 22:05:50.622789 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.622764 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db"] Jul 28 22:05:50.640004 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.639973 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w"] Jul 28 22:05:50.642056 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.642040 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.644607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.644587 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 22:05:50.644851 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.644833 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 22:05:50.645148 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.645127 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 22:05:50.645302 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.645285 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2c5ds\"" Jul 28 22:05:50.658065 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.658035 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sqkz6"] Jul 28 22:05:50.660665 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.660644 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w"] Jul 28 22:05:50.660808 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.660737 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.663406 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.663385 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 22:05:50.663540 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.663526 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 22:05:50.663630 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.663612 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 22:05:50.665325 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665299 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.665420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665331 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmh7q\" (UniqueName: \"kubernetes.io/projected/4f94d17c-8df6-4982-b6a7-f2b2d2711769-kube-api-access-nmh7q\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.665420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665358 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9km9m\" (UniqueName: \"kubernetes.io/projected/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-api-access-9km9m\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.665420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665399 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f94d17c-8df6-4982-b6a7-f2b2d2711769-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.665569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665427 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/704c6d24-83d0-421d-a2a2-d1d33be6a108-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.665569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665460 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.665569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665477 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f94d17c-8df6-4982-b6a7-f2b2d2711769-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.665569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665505 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.665569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665529 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gsb4z\"" Jul 28 22:05:50.665569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665534 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f94d17c-8df6-4982-b6a7-f2b2d2711769-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.665793 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.665602 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/704c6d24-83d0-421d-a2a2-d1d33be6a108-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.766595 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766557 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/704c6d24-83d0-421d-a2a2-d1d33be6a108-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.766595 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766598 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-sys\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.766791 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766617 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-metrics-client-ca\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.766791 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766652 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.766791 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766676 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmh7q\" (UniqueName: \"kubernetes.io/projected/4f94d17c-8df6-4982-b6a7-f2b2d2711769-kube-api-access-nmh7q\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.766791 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.766791 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766742 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.766791 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766774 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9km9m\" (UniqueName: \"kubernetes.io/projected/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-api-access-9km9m\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.767094 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766800 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f94d17c-8df6-4982-b6a7-f2b2d2711769-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.767094 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/704c6d24-83d0-421d-a2a2-d1d33be6a108-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.767094 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766888 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-textfile\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.767094 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766920 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.767094 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766970 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f94d17c-8df6-4982-b6a7-f2b2d2711769-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.767094 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.766986 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/704c6d24-83d0-421d-a2a2-d1d33be6a108-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.767094 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.767006 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-root\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.767094 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.767045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.767094 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:50.767063 2557 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 28 22:05:50.767498 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.767068 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-tls\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.767498 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:50.767149 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-tls podName:704c6d24-83d0-421d-a2a2-d1d33be6a108 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:51.267104094 +0000 UTC m=+75.904963217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-hnp4w" (UID: "704c6d24-83d0-421d-a2a2-d1d33be6a108") : secret "kube-state-metrics-tls" not found Jul 28 22:05:50.767498 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.767211 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f94d17c-8df6-4982-b6a7-f2b2d2711769-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.767498 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.767242 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-wtmp\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.767498 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.767270 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95j6\" (UniqueName: \"kubernetes.io/projected/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-kube-api-access-n95j6\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.767498 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.767414 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.767706 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.767607 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/704c6d24-83d0-421d-a2a2-d1d33be6a108-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.767706 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.767654 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f94d17c-8df6-4982-b6a7-f2b2d2711769-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.769449 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.769428 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f94d17c-8df6-4982-b6a7-f2b2d2711769-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.769548 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.769433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.769548 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.769489 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f94d17c-8df6-4982-b6a7-f2b2d2711769-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.777862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.777830 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmh7q\" (UniqueName: \"kubernetes.io/projected/4f94d17c-8df6-4982-b6a7-f2b2d2711769-kube-api-access-nmh7q\") pod \"openshift-state-metrics-b8f49f5c6-ts6db\" (UID: \"4f94d17c-8df6-4982-b6a7-f2b2d2711769\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:50.778499 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.778475 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9km9m\" (UniqueName: \"kubernetes.io/projected/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-api-access-9km9m\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:50.868265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868175 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-sys\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-metrics-client-ca\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868236 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868490 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868300 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-sys\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868490 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868347 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868490 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868378 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-textfile\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868641 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868608 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-root\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868698 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868667 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-tls\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868698 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868676 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-root\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868790 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868717 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-textfile\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868790 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868726 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-wtmp\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868790 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868758 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95j6\" (UniqueName: \"kubernetes.io/projected/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-kube-api-access-n95j6\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868953 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:50.868789 2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 22:05:50.868953 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868843 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-metrics-client-ca\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868953 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868851 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-wtmp\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.868953 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:50.868852 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-tls podName:885e99b9-f1b7-4f03-a8a6-0f005b08bf34 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:51.368831801 +0000 UTC m=+76.006690925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-tls") pod "node-exporter-sqkz6" (UID: "885e99b9-f1b7-4f03-a8a6-0f005b08bf34") : secret "node-exporter-tls" not found Jul 28 22:05:50.868953 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.868907 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.870509 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.870491 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.881213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.881191 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95j6\" (UniqueName: \"kubernetes.io/projected/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-kube-api-access-n95j6\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:50.921179 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:50.921146 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" Jul 28 22:05:51.047365 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.047302 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db"] Jul 28 22:05:51.051023 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:51.050994 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f94d17c_8df6_4982_b6a7_f2b2d2711769.slice/crio-2e90819ee68763f0fc0a5285205dc63679e220442182e047d3ac48065c00e712 WatchSource:0}: Error finding container 2e90819ee68763f0fc0a5285205dc63679e220442182e047d3ac48065c00e712: Status 404 returned error can't find the container with id 2e90819ee68763f0fc0a5285205dc63679e220442182e047d3ac48065c00e712 Jul 28 22:05:51.273039 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.272993 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:51.275376 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.275350 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/704c6d24-83d0-421d-a2a2-d1d33be6a108-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-hnp4w\" (UID: \"704c6d24-83d0-421d-a2a2-d1d33be6a108\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:51.374392 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.374356 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-tls\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:51.376688 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.376670 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/885e99b9-f1b7-4f03-a8a6-0f005b08bf34-node-exporter-tls\") pod \"node-exporter-sqkz6\" (UID: \"885e99b9-f1b7-4f03-a8a6-0f005b08bf34\") " pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:51.497242 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.497214 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" event={"ID":"4f94d17c-8df6-4982-b6a7-f2b2d2711769","Type":"ContainerStarted","Data":"4114967bbad751959a256231c087e58b5675bfd67fe262f5942ea85598587163"} Jul 28 22:05:51.497400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.497267 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" event={"ID":"4f94d17c-8df6-4982-b6a7-f2b2d2711769","Type":"ContainerStarted","Data":"103e65c7e1ca00c0a9c58ec30f27b826798f0fff64635736ff074f09bcf613f9"} Jul 28 22:05:51.497400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.497284 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" event={"ID":"4f94d17c-8df6-4982-b6a7-f2b2d2711769","Type":"ContainerStarted","Data":"2e90819ee68763f0fc0a5285205dc63679e220442182e047d3ac48065c00e712"} Jul 28 22:05:51.551601 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.551570 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" Jul 28 22:05:51.570349 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.570305 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sqkz6" Jul 28 22:05:51.581608 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:51.581581 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885e99b9_f1b7_4f03_a8a6_0f005b08bf34.slice/crio-d1fb5276e1a099015d5fe58ddc05df46b0f50bf700b33a1fcbcb7e4ace96cf25 WatchSource:0}: Error finding container d1fb5276e1a099015d5fe58ddc05df46b0f50bf700b33a1fcbcb7e4ace96cf25: Status 404 returned error can't find the container with id d1fb5276e1a099015d5fe58ddc05df46b0f50bf700b33a1fcbcb7e4ace96cf25 Jul 28 22:05:51.711694 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.711652 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 22:05:51.714793 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.714763 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.714909 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.714858 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w"] Jul 28 22:05:51.716491 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:51.716448 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704c6d24_83d0_421d_a2a2_d1d33be6a108.slice/crio-1f6e4c2f7a3bc1d2105cb6d17fa8580a7c370ad37e678d0478b6bc07f3111c4d WatchSource:0}: Error finding container 1f6e4c2f7a3bc1d2105cb6d17fa8580a7c370ad37e678d0478b6bc07f3111c4d: Status 404 returned error can't find the container with id 1f6e4c2f7a3bc1d2105cb6d17fa8580a7c370ad37e678d0478b6bc07f3111c4d Jul 28 22:05:51.718151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718132 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 22:05:51.718151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718146 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 22:05:51.718286 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718174 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 22:05:51.718286 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718195 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 22:05:51.718286 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718238 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 22:05:51.718286 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718139 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 22:05:51.718631 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718291 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 22:05:51.718631 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718304 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 22:05:51.718631 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718340 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 22:05:51.718631 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.718384 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ktxwf\"" Jul 28 22:05:51.729175 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.729152 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 22:05:51.778902 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.778817 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-config-out\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.778902 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.778862 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779143 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779006 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779143 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779039 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779143 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779076 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779143 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779115 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779285 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779170 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-web-config\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779285 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779191 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779285 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779260 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779281 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779352 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhfv\" (UniqueName: \"kubernetes.io/projected/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-kube-api-access-fhhfv\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779381 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.779420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.779406 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.879918 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.879867 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.879938 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.879978 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhfv\" (UniqueName: \"kubernetes.io/projected/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-kube-api-access-fhhfv\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880034 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880087 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-config-out\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880118 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880155 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880183 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880426 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:51.880217 2557 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 28 22:05:51.880426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880426 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:51.880274 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-main-tls podName:b6b4e66e-1e5d-4b5e-8fe1-97095194fd69 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:52.380257381 +0000 UTC m=+77.018116488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b6b4e66e-1e5d-4b5e-8fe1-97095194fd69") : secret "alertmanager-main-tls" not found Jul 28 22:05:51.880426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-web-config\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.880426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.880324 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.883309 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:51.882992 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-alertmanager-trusted-ca-bundle podName:b6b4e66e-1e5d-4b5e-8fe1-97095194fd69 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:52.382971226 +0000 UTC m=+77.020830347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "b6b4e66e-1e5d-4b5e-8fe1-97095194fd69") : configmap references non-existent config key: ca-bundle.crt Jul 28 22:05:51.883309 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.883203 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.883309 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.883272 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.883579 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.883553 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.883809 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.883779 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.884307 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.883979 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-web-config\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.884307 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.884047 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-config-out\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.884307 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.884295 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.884642 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.884402 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.884848 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.884826 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.885785 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.885741 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:51.890248 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:51.890227 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhfv\" (UniqueName: \"kubernetes.io/projected/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-kube-api-access-fhhfv\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:52.384292 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.384203 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:52.384465 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.384356 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:52.385547 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.385515 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:52.389329 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.387408 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6b4e66e-1e5d-4b5e-8fe1-97095194fd69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:52.503317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.503278 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqkz6" event={"ID":"885e99b9-f1b7-4f03-a8a6-0f005b08bf34","Type":"ContainerStarted","Data":"d1fb5276e1a099015d5fe58ddc05df46b0f50bf700b33a1fcbcb7e4ace96cf25"} Jul 28 22:05:52.504730 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.504696 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" event={"ID":"704c6d24-83d0-421d-a2a2-d1d33be6a108","Type":"ContainerStarted","Data":"1f6e4c2f7a3bc1d2105cb6d17fa8580a7c370ad37e678d0478b6bc07f3111c4d"} Jul 28 22:05:52.507217 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.507190 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" event={"ID":"4f94d17c-8df6-4982-b6a7-f2b2d2711769","Type":"ContainerStarted","Data":"a9be3ebe159f40053f1c3240a0ed8972d2c56225a9b8f29436e68012a1541cd2"} Jul 28 22:05:52.607584 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.607520 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ts6db" podStartSLOduration=1.631760691 podStartE2EDuration="2.60749643s" totalImagesPullingTime="975.735739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:52.528129915 +0000 UTC m=+77.165989044" watchObservedRunningTime="2026-07-28 22:05:52.60749643 +0000 UTC m=+77.245355558" Jul 28 22:05:52.608734 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.608713 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76f96b9bf6-fjq27"] Jul 28 22:05:52.613145 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.613118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.615984 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.615862 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 22:05:52.615984 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.615862 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 22:05:52.616169 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.616040 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4jvvv\"" Jul 28 22:05:52.616169 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.616046 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 22:05:52.616169 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.616080 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 22:05:52.616774 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.616616 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 22:05:52.616774 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.616619 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1fkmjo2le2tvb\"" Jul 28 22:05:52.624363 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.624344 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76f96b9bf6-fjq27"] Jul 28 22:05:52.628856 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.628812 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 22:05:52.687584 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.687492 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b2106be-fee7-49ef-9fff-455c07ab8185-metrics-client-ca\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.687584 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.687556 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.687818 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.687612 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.687818 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.687638 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.687818 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.687683 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.687818 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.687707 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-grpc-tls\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.687818 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.687730 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zzs6\" (UniqueName: \"kubernetes.io/projected/9b2106be-fee7-49ef-9fff-455c07ab8185-kube-api-access-8zzs6\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.687818 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.687756 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-tls\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.789360 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.789327 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.790151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.789401 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.790151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.789433 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.790151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.789479 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.790151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.789508 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-grpc-tls\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.790151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.789536 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zzs6\" (UniqueName: \"kubernetes.io/projected/9b2106be-fee7-49ef-9fff-455c07ab8185-kube-api-access-8zzs6\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.790151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.789565 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-tls\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.790151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.789624 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b2106be-fee7-49ef-9fff-455c07ab8185-metrics-client-ca\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.790531 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.790377 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b2106be-fee7-49ef-9fff-455c07ab8185-metrics-client-ca\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.793141 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.793093 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.793705 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.793425 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.793705 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.793664 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.793880 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.793858 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.794578 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.794538 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-grpc-tls\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.795957 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.795899 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9b2106be-fee7-49ef-9fff-455c07ab8185-secret-thanos-querier-tls\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.798715 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.798692 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zzs6\" (UniqueName: \"kubernetes.io/projected/9b2106be-fee7-49ef-9fff-455c07ab8185-kube-api-access-8zzs6\") pod \"thanos-querier-76f96b9bf6-fjq27\" (UID: \"9b2106be-fee7-49ef-9fff-455c07ab8185\") " pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:52.871250 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.871216 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 22:05:52.926064 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:52.926033 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:53.153577 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:53.153537 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b4e66e_1e5d_4b5e_8fe1_97095194fd69.slice/crio-ff3dae0073634f9eaf7f96d8e984fa48283afbbfddb43c72c43dfa04b0c232e7 WatchSource:0}: Error finding container ff3dae0073634f9eaf7f96d8e984fa48283afbbfddb43c72c43dfa04b0c232e7: Status 404 returned error can't find the container with id ff3dae0073634f9eaf7f96d8e984fa48283afbbfddb43c72c43dfa04b0c232e7 Jul 28 22:05:53.288419 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:53.288329 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76f96b9bf6-fjq27"] Jul 28 22:05:53.290384 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:53.290352 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2106be_fee7_49ef_9fff_455c07ab8185.slice/crio-abe3bc0cebdb74b52ecc67f74e5b77ba565f2193767f97646100c5749bfe8933 WatchSource:0}: Error finding container abe3bc0cebdb74b52ecc67f74e5b77ba565f2193767f97646100c5749bfe8933: Status 404 returned error can't find the container with id abe3bc0cebdb74b52ecc67f74e5b77ba565f2193767f97646100c5749bfe8933 Jul 28 22:05:53.515879 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:53.515834 2557 generic.go:356] "Generic (PLEG): container finished" podID="885e99b9-f1b7-4f03-a8a6-0f005b08bf34" containerID="ea3414fe8233b6a50f27cb513ae5a53c8ef2fc840d35aa9dccdf16acaf348ecc" exitCode=0 Jul 28 22:05:53.516108 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:53.515881 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqkz6" event={"ID":"885e99b9-f1b7-4f03-a8a6-0f005b08bf34","Type":"ContainerDied","Data":"ea3414fe8233b6a50f27cb513ae5a53c8ef2fc840d35aa9dccdf16acaf348ecc"} Jul 28 22:05:53.518688 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:53.518663 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" event={"ID":"704c6d24-83d0-421d-a2a2-d1d33be6a108","Type":"ContainerStarted","Data":"704d1cc613a538a7b459d151a6a1e371e2d2a8efbb509d457d34c095138b5160"} Jul 28 22:05:53.518803 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:53.518716 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" event={"ID":"704c6d24-83d0-421d-a2a2-d1d33be6a108","Type":"ContainerStarted","Data":"d433d34265e17918715b3c51235aba5eb753bf8446042cf6cb30f30fbe058d35"} Jul 28 22:05:53.518803 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:53.518746 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" event={"ID":"704c6d24-83d0-421d-a2a2-d1d33be6a108","Type":"ContainerStarted","Data":"b41cf6c9c2c3554d031f8d825df9094003bb7d066c2ce9ebf12fe2f1e81fead1"} Jul 28 22:05:53.521880 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:53.521856 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69","Type":"ContainerStarted","Data":"ff3dae0073634f9eaf7f96d8e984fa48283afbbfddb43c72c43dfa04b0c232e7"} Jul 28 22:05:53.523373 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:53.523352 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" event={"ID":"9b2106be-fee7-49ef-9fff-455c07ab8185","Type":"ContainerStarted","Data":"abe3bc0cebdb74b52ecc67f74e5b77ba565f2193767f97646100c5749bfe8933"} Jul 28 22:05:53.552858 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:53.552812 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-hnp4w" podStartSLOduration=2.089690929 podStartE2EDuration="3.552794244s" totalImagesPullingTime="1.463103315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:53.55143187 +0000 UTC m=+78.189291000" watchObservedRunningTime="2026-07-28 22:05:53.552794244 +0000 UTC m=+78.190653380" Jul 28 22:05:54.531033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:54.530990 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqkz6" event={"ID":"885e99b9-f1b7-4f03-a8a6-0f005b08bf34","Type":"ContainerStarted","Data":"c551f25f308eb691282434bb0e1710aec0d0b5685ee6965df2536df36ec74e55"} Jul 28 22:05:54.531499 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:54.531098 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqkz6" event={"ID":"885e99b9-f1b7-4f03-a8a6-0f005b08bf34","Type":"ContainerStarted","Data":"85efe3471d180f14a444c2ebb952b7a650068c666ed7060196cf738ae5d74325"} Jul 28 22:05:54.552160 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:54.552096 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sqkz6" podStartSLOduration=3.323301059 podStartE2EDuration="4.552074407s" totalImagesPullingTime="1.15874354s" totalInitContainerRuntime="70.029808ms" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:54.550819434 +0000 UTC m=+79.188678563" watchObservedRunningTime="2026-07-28 22:05:54.552074407 +0000 UTC m=+79.189933540" Jul 28 22:05:55.054953 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.054863 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bc7996b9-g9q79"] Jul 28 22:05:55.058046 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.058020 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.062665 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.062624 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qpgp2\"" Jul 28 22:05:55.062665 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.062624 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-esopqk5slpilu\"" Jul 28 22:05:55.062851 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.062687 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 22:05:55.062851 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.062729 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 28 22:05:55.063001 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.062979 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 28 22:05:55.063001 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.062983 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 28 22:05:55.066341 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.066279 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bc7996b9-g9q79"] Jul 28 22:05:55.110089 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.110049 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/01cfad16-d765-4b87-9ea4-8fdc52705c50-secret-metrics-server-client-certs\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.110265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.110096 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q48b\" (UniqueName: \"kubernetes.io/projected/01cfad16-d765-4b87-9ea4-8fdc52705c50-kube-api-access-8q48b\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.110265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.110164 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/01cfad16-d765-4b87-9ea4-8fdc52705c50-client-ca-bundle\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.110265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.110182 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/01cfad16-d765-4b87-9ea4-8fdc52705c50-metrics-server-audit-profiles\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.110265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.110206 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/01cfad16-d765-4b87-9ea4-8fdc52705c50-audit-log\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.110265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.110236 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01cfad16-d765-4b87-9ea4-8fdc52705c50-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.110448 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.110301 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/01cfad16-d765-4b87-9ea4-8fdc52705c50-secret-metrics-server-tls\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.211687 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.211655 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/01cfad16-d765-4b87-9ea4-8fdc52705c50-secret-metrics-server-tls\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.211865 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.211702 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/01cfad16-d765-4b87-9ea4-8fdc52705c50-secret-metrics-server-client-certs\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.211865 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.211729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q48b\" (UniqueName: \"kubernetes.io/projected/01cfad16-d765-4b87-9ea4-8fdc52705c50-kube-api-access-8q48b\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.211865 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.211781 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/01cfad16-d765-4b87-9ea4-8fdc52705c50-client-ca-bundle\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.211865 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.211805 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/01cfad16-d765-4b87-9ea4-8fdc52705c50-metrics-server-audit-profiles\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.211865 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.211831 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/01cfad16-d765-4b87-9ea4-8fdc52705c50-audit-log\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.212166 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.211868 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01cfad16-d765-4b87-9ea4-8fdc52705c50-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.212776 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.212691 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/01cfad16-d765-4b87-9ea4-8fdc52705c50-audit-log\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.212963 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.212883 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01cfad16-d765-4b87-9ea4-8fdc52705c50-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.213357 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.213332 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/01cfad16-d765-4b87-9ea4-8fdc52705c50-metrics-server-audit-profiles\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.214649 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.214625 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/01cfad16-d765-4b87-9ea4-8fdc52705c50-secret-metrics-server-tls\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.214752 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.214698 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/01cfad16-d765-4b87-9ea4-8fdc52705c50-secret-metrics-server-client-certs\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.214807 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.214774 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/01cfad16-d765-4b87-9ea4-8fdc52705c50-client-ca-bundle\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.224907 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.224883 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q48b\" (UniqueName: \"kubernetes.io/projected/01cfad16-d765-4b87-9ea4-8fdc52705c50-kube-api-access-8q48b\") pod \"metrics-server-bc7996b9-g9q79\" (UID: \"01cfad16-d765-4b87-9ea4-8fdc52705c50\") " pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.370010 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.369920 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:05:55.413758 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.413726 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4"] Jul 28 22:05:55.417183 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.417160 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:55.419518 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.419497 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 22:05:55.419610 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.419548 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k64v2\"" Jul 28 22:05:55.426198 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.426175 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4"] Jul 28 22:05:55.514772 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.514743 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmcqh\" (UniqueName: \"kubernetes.io/projected/3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7-kube-api-access-fmcqh\") pod \"monitoring-plugin-68b7fc9574-5g7v4\" (UID: \"3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:55.514992 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.514785 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-5g7v4\" (UID: \"3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:55.537978 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.537939 2557 generic.go:356] "Generic (PLEG): container finished" podID="b6b4e66e-1e5d-4b5e-8fe1-97095194fd69" containerID="4fc83bd620ece7765f53aca582e2ac7c7105db5f0e4c99f992a19dfe3d505047" exitCode=0 Jul 28 22:05:55.538338 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.538028 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69","Type":"ContainerDied","Data":"4fc83bd620ece7765f53aca582e2ac7c7105db5f0e4c99f992a19dfe3d505047"} Jul 28 22:05:55.616677 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.616264 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmcqh\" (UniqueName: \"kubernetes.io/projected/3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7-kube-api-access-fmcqh\") pod \"monitoring-plugin-68b7fc9574-5g7v4\" (UID: \"3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:55.616677 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.616396 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-5g7v4\" (UID: \"3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:55.617252 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:55.617228 2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 28 22:05:55.617371 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:05:55.617302 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7-monitoring-plugin-cert podName:3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:56.117279271 +0000 UTC m=+80.755138381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-5g7v4" (UID: "3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7") : secret "monitoring-plugin-cert" not found Jul 28 22:05:55.626607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.626584 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmcqh\" (UniqueName: \"kubernetes.io/projected/3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7-kube-api-access-fmcqh\") pod \"monitoring-plugin-68b7fc9574-5g7v4\" (UID: \"3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:55.702897 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:55.702877 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bc7996b9-g9q79"] Jul 28 22:05:55.705597 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:55.705574 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cfad16_d765_4b87_9ea4_8fdc52705c50.slice/crio-d2eb4ddaa6a834588c9c38e1a24cf61d89be39052e70c18ca0d1b9fe86abcc98 WatchSource:0}: Error finding container d2eb4ddaa6a834588c9c38e1a24cf61d89be39052e70c18ca0d1b9fe86abcc98: Status 404 returned error can't find the container with id d2eb4ddaa6a834588c9c38e1a24cf61d89be39052e70c18ca0d1b9fe86abcc98 Jul 28 22:05:56.119912 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.119842 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-5g7v4\" (UID: \"3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:56.122302 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.122278 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-5g7v4\" (UID: \"3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:56.332176 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.329484 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:56.500812 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.500732 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4"] Jul 28 22:05:56.507257 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:56.507210 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0f08eb_98cc_47f6_b083_9a2bebbbbaa7.slice/crio-c3f47ee24b6a835d4342fd3b88767f79bfc0095f9b1e6a95732fdefe0d02c12b WatchSource:0}: Error finding container c3f47ee24b6a835d4342fd3b88767f79bfc0095f9b1e6a95732fdefe0d02c12b: Status 404 returned error can't find the container with id c3f47ee24b6a835d4342fd3b88767f79bfc0095f9b1e6a95732fdefe0d02c12b Jul 28 22:05:56.546100 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.545899 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" event={"ID":"9b2106be-fee7-49ef-9fff-455c07ab8185","Type":"ContainerStarted","Data":"1024ccf1d72986bc3376c33d4b44790f4d73af585c65b8e4da1a2d1d56be8584"} Jul 28 22:05:56.546100 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.545999 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" event={"ID":"9b2106be-fee7-49ef-9fff-455c07ab8185","Type":"ContainerStarted","Data":"7842a271e8d1ce504efa6c31b67d85587d3efce1b33e83935f839c3c850c2ab2"} Jul 28 22:05:56.546100 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.546042 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" event={"ID":"9b2106be-fee7-49ef-9fff-455c07ab8185","Type":"ContainerStarted","Data":"bb0f30b3b57fb34bcdb85ebf21974d5777cec40a0bf55c659a635f8d7d0a51b6"} Jul 28 22:05:56.547729 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.547662 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" event={"ID":"01cfad16-d765-4b87-9ea4-8fdc52705c50","Type":"ContainerStarted","Data":"d2eb4ddaa6a834588c9c38e1a24cf61d89be39052e70c18ca0d1b9fe86abcc98"} Jul 28 22:05:56.549465 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.549411 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" event={"ID":"3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7","Type":"ContainerStarted","Data":"c3f47ee24b6a835d4342fd3b88767f79bfc0095f9b1e6a95732fdefe0d02c12b"} Jul 28 22:05:56.917867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.917829 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 22:05:56.925819 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.925797 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:56.930197 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.929977 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 22:05:56.930197 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.930123 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 22:05:56.930197 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.930191 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 22:05:56.930468 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.930317 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vbnf4\"" Jul 28 22:05:56.933213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.932610 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 22:05:56.933213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.932777 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 22:05:56.933213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.932825 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 22:05:56.933213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.932877 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 22:05:56.933213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.932826 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 22:05:56.933213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.933046 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 22:05:56.933213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.933065 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 22:05:56.933647 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.933378 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 22:05:56.934033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.934009 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 22:05:56.934666 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.934406 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-710900hll8kgc\"" Jul 28 22:05:56.935635 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.935617 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 22:05:56.936204 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.936164 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 22:05:56.941371 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:56.941282 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 22:05:57.026813 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.026776 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.026813 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.026815 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027093 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.026854 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027093 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.026895 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027093 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.026946 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027093 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.026964 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027093 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027026 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-config\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027093 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027059 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027114 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027147 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-config-out\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027174 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-web-config\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027198 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027222 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.027310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027290 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.029805 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027321 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.029805 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027350 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.029805 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027375 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.029805 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027398 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5rw\" (UniqueName: \"kubernetes.io/projected/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-kube-api-access-2h5rw\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.029805 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027420 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.029805 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.027462 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.132018 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.131975 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.132216 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132027 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.132216 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132060 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-config\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.132216 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132611 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132650 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-config-out\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132685 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-web-config\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132711 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132771 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132803 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132847 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132872 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.132858 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.133276 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5rw\" (UniqueName: \"kubernetes.io/projected/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-kube-api-access-2h5rw\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.133308 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.133343 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.133405 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.133437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.133503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.133481 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.134258 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.133534 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.134258 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.134034 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.134569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.134542 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.135620 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.135300 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.135620 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.135497 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.136557 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.136257 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.136557 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.136350 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.137592 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.137567 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.138493 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.138002 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.138493 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.138456 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.139125 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.139087 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.139660 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.139589 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-config-out\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.140522 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.140496 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.140700 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.140653 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.140919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.140874 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-web-config\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.141444 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.141404 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-config\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.142496 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.142469 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.143481 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.143441 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5rw\" (UniqueName: \"kubernetes.io/projected/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-kube-api-access-2h5rw\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.143634 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.143618 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06c336d7-bb1f-4b8f-9a0f-fd6118931a14-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06c336d7-bb1f-4b8f-9a0f-fd6118931a14\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.240630 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.240598 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:05:57.832128 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:57.831694 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 22:05:57.833048 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:05:57.832699 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c336d7_bb1f_4b8f_9a0f_fd6118931a14.slice/crio-50c88a09bd6de420efd8be0c7e3edcf524f0f7a8a107555d3ebc5e839ebb9347 WatchSource:0}: Error finding container 50c88a09bd6de420efd8be0c7e3edcf524f0f7a8a107555d3ebc5e839ebb9347: Status 404 returned error can't find the container with id 50c88a09bd6de420efd8be0c7e3edcf524f0f7a8a107555d3ebc5e839ebb9347 Jul 28 22:05:58.563119 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.563066 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69","Type":"ContainerStarted","Data":"4a8661b3941a9b60365d590349bb058a74afef417d9ae44bee625903003de57b"} Jul 28 22:05:58.563450 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.563148 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69","Type":"ContainerStarted","Data":"a539bec9e87e6d40a4e29800ab44180af6d9df1da47c83dbe57d90ce31110bd0"} Jul 28 22:05:58.563450 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.563186 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69","Type":"ContainerStarted","Data":"1785f8c3d461600bed8128424aa073b825cf4232ff333c326c42189e626978cb"} Jul 28 22:05:58.563450 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.563223 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69","Type":"ContainerStarted","Data":"edfea68432a76e1b0ff4e2d068c365d2f7e6c9f305effac630ff3889269abc9c"} Jul 28 22:05:58.563450 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.563256 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69","Type":"ContainerStarted","Data":"d8cb8c533d70b6f0b124718ba19fa93cea6b0e658c314f69cd4e72eb25380e23"} Jul 28 22:05:58.563450 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.563290 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6b4e66e-1e5d-4b5e-8fe1-97095194fd69","Type":"ContainerStarted","Data":"87cb5e61c8972ac6ec06b9f843c5e772fed3065dc1b733b76ec40ac57673eb5c"} Jul 28 22:05:58.566743 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.566704 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" event={"ID":"9b2106be-fee7-49ef-9fff-455c07ab8185","Type":"ContainerStarted","Data":"3513206339dc6b260f936d00b0810103d120d3042ce1d0e505f630caafbd770d"} Jul 28 22:05:58.566862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.566772 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" event={"ID":"9b2106be-fee7-49ef-9fff-455c07ab8185","Type":"ContainerStarted","Data":"59e5d7b06b2228a7ecc2bebfd0a7a35038a1c7942edbd88845cccf842654ab45"} Jul 28 22:05:58.566862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.566815 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:05:58.566862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.566852 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" event={"ID":"9b2106be-fee7-49ef-9fff-455c07ab8185","Type":"ContainerStarted","Data":"f947cb86abf305676fc5d0571f598826f95d88041bf74c008ca326874bfb07d4"} Jul 28 22:05:58.568865 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.568834 2557 generic.go:356] "Generic (PLEG): container finished" podID="06c336d7-bb1f-4b8f-9a0f-fd6118931a14" containerID="89dbbe9f5778135d73edd8c0886db9d9869d2ec45a5451728e8f0b4e4e75863e" exitCode=0 Jul 28 22:05:58.568999 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.568941 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06c336d7-bb1f-4b8f-9a0f-fd6118931a14","Type":"ContainerDied","Data":"89dbbe9f5778135d73edd8c0886db9d9869d2ec45a5451728e8f0b4e4e75863e"} Jul 28 22:05:58.569067 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.569013 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06c336d7-bb1f-4b8f-9a0f-fd6118931a14","Type":"ContainerStarted","Data":"50c88a09bd6de420efd8be0c7e3edcf524f0f7a8a107555d3ebc5e839ebb9347"} Jul 28 22:05:58.571852 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.571831 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" event={"ID":"01cfad16-d765-4b87-9ea4-8fdc52705c50","Type":"ContainerStarted","Data":"22e16484df583351f3dac6885cbfc4bd31ff925307de584cd8ab3d422ad9bbdd"} Jul 28 22:05:58.590511 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.590423 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.893214408 podStartE2EDuration="7.590401027s" totalImagesPullingTime="3.682863633s" totalInitContainerRuntime="14.322986ms" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:58.58876473 +0000 UTC m=+83.226623885" watchObservedRunningTime="2026-07-28 22:05:58.590401027 +0000 UTC m=+83.228260157" Jul 28 22:05:58.615625 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:58.614250 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" podStartSLOduration=2.434811328 podStartE2EDuration="6.614228408s" totalImagesPullingTime="4.17941708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:58.611147755 +0000 UTC m=+83.249006882" watchObservedRunningTime="2026-07-28 22:05:58.614228408 +0000 UTC m=+83.252087542" Jul 28 22:05:59.577550 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:59.577507 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" event={"ID":"3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7","Type":"ContainerStarted","Data":"1100e58043e12cfed505795c0cd148cf7d21392c8f5b541513e92fcdfaa68b35"} Jul 28 22:05:59.578244 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:59.578227 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:59.582550 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:59.582530 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" Jul 28 22:05:59.592682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:59.592638 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5g7v4" podStartSLOduration=2.407206235 podStartE2EDuration="4.592622593s" totalImagesPullingTime="2.185416358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:59.592232118 +0000 UTC m=+84.230091247" watchObservedRunningTime="2026-07-28 22:05:59.592622593 +0000 UTC m=+84.230481789" Jul 28 22:05:59.592955 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:05:59.592918 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" podStartSLOduration=2.632052312 podStartE2EDuration="4.592912062s" totalImagesPullingTime="1.96085975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:58.679471388 +0000 UTC m=+83.317330516" watchObservedRunningTime="2026-07-28 22:05:59.592912062 +0000 UTC m=+84.230771191" Jul 28 22:06:02.594021 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:02.593989 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06c336d7-bb1f-4b8f-9a0f-fd6118931a14","Type":"ContainerStarted","Data":"f90dd92ff6ea7a4b9d277e6af425c6255a03abaeedd9395a847d0aabe89de269"} Jul 28 22:06:03.602085 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:03.602047 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06c336d7-bb1f-4b8f-9a0f-fd6118931a14","Type":"ContainerStarted","Data":"4aae1fb4940d3ea8f4fa98ab514812cdde0b4415784d0ffeb2a546b372c36346"} Jul 28 22:06:03.602496 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:03.602124 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06c336d7-bb1f-4b8f-9a0f-fd6118931a14","Type":"ContainerStarted","Data":"c9f6d761070aa4625186719611815c4ec7b4dc609abaa00dd4bcc07d5b9fe5ef"} Jul 28 22:06:03.602496 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:03.602166 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06c336d7-bb1f-4b8f-9a0f-fd6118931a14","Type":"ContainerStarted","Data":"9b0f7d154b5e3fbcbe266de0fb563efa914ceb049bf79709edef7ec9dd0b8790"} Jul 28 22:06:03.602496 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:03.602203 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06c336d7-bb1f-4b8f-9a0f-fd6118931a14","Type":"ContainerStarted","Data":"b49940d5e748753598cd8b1d72fafa2aa28dff15237d415018962697340d85db"} Jul 28 22:06:03.602496 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:03.602235 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06c336d7-bb1f-4b8f-9a0f-fd6118931a14","Type":"ContainerStarted","Data":"de3a567ed5827b0a3f8e14ccc3466e42bf4465d8042b6005537b9b42fd5603aa"} Jul 28 22:06:03.631252 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:03.631195 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.7024606909999997 podStartE2EDuration="7.631178028s" totalImagesPullingTime="3.896339417s" totalInitContainerRuntime="32.37792ms" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:06:03.629007739 +0000 UTC m=+88.266866869" watchObservedRunningTime="2026-07-28 22:06:03.631178028 +0000 UTC m=+88.269037155" Jul 28 22:06:04.583700 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:04.583667 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76f96b9bf6-fjq27" Jul 28 22:06:07.241640 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:07.241607 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:06:08.119970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:08.119920 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf8854547-qcp2f"] Jul 28 22:06:09.491245 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.491181 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-776d948f57-5692h" podUID="16902eed-a0c9-4180-8b79-3cb0795cd6d9" containerName="console" containerID="cri-o://20d60ff2c724abebe6a8210a89628b0de2a5cc7bac006f6428b2721e3463feed" gracePeriod=15 Jul 28 22:06:09.630618 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.630587 2557 generic.go:356] "Generic (PLEG): container finished" podID="16902eed-a0c9-4180-8b79-3cb0795cd6d9" containerID="20d60ff2c724abebe6a8210a89628b0de2a5cc7bac006f6428b2721e3463feed" exitCode=0 Jul 28 22:06:09.630788 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.630642 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776d948f57-5692h" event={"ID":"16902eed-a0c9-4180-8b79-3cb0795cd6d9","Type":"ContainerDied","Data":"20d60ff2c724abebe6a8210a89628b0de2a5cc7bac006f6428b2721e3463feed"} Jul 28 22:06:09.731452 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.731431 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776d948f57-5692h" Jul 28 22:06:09.856861 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.856770 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-oauth-config\") pod \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " Jul 28 22:06:09.856861 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.856814 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-oauth-serving-cert\") pod \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " Jul 28 22:06:09.856861 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.856844 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-serving-cert\") pod \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " Jul 28 22:06:09.857200 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.856902 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p77q\" (UniqueName: \"kubernetes.io/projected/16902eed-a0c9-4180-8b79-3cb0795cd6d9-kube-api-access-4p77q\") pod \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " Jul 28 22:06:09.857200 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.856957 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-config\") pod \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " Jul 28 22:06:09.857200 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.856976 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-service-ca\") pod \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\" (UID: \"16902eed-a0c9-4180-8b79-3cb0795cd6d9\") " Jul 28 22:06:09.857400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.857373 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "16902eed-a0c9-4180-8b79-3cb0795cd6d9" (UID: "16902eed-a0c9-4180-8b79-3cb0795cd6d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:06:09.857460 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.857390 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-config" (OuterVolumeSpecName: "console-config") pod "16902eed-a0c9-4180-8b79-3cb0795cd6d9" (UID: "16902eed-a0c9-4180-8b79-3cb0795cd6d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:06:09.857460 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.857417 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "16902eed-a0c9-4180-8b79-3cb0795cd6d9" (UID: "16902eed-a0c9-4180-8b79-3cb0795cd6d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:06:09.859288 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.859259 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "16902eed-a0c9-4180-8b79-3cb0795cd6d9" (UID: "16902eed-a0c9-4180-8b79-3cb0795cd6d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:06:09.859556 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.859530 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "16902eed-a0c9-4180-8b79-3cb0795cd6d9" (UID: "16902eed-a0c9-4180-8b79-3cb0795cd6d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:06:09.859653 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.859572 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16902eed-a0c9-4180-8b79-3cb0795cd6d9-kube-api-access-4p77q" (OuterVolumeSpecName: "kube-api-access-4p77q") pod "16902eed-a0c9-4180-8b79-3cb0795cd6d9" (UID: "16902eed-a0c9-4180-8b79-3cb0795cd6d9"). InnerVolumeSpecName "kube-api-access-4p77q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:06:09.958208 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.958171 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-oauth-config\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:09.958208 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.958198 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-oauth-serving-cert\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:09.958208 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.958207 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-serving-cert\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:09.958208 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.958216 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p77q\" (UniqueName: \"kubernetes.io/projected/16902eed-a0c9-4180-8b79-3cb0795cd6d9-kube-api-access-4p77q\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:09.958466 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.958225 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-console-config\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:09.958466 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:09.958233 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16902eed-a0c9-4180-8b79-3cb0795cd6d9-service-ca\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:10.637115 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:10.637076 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776d948f57-5692h" event={"ID":"16902eed-a0c9-4180-8b79-3cb0795cd6d9","Type":"ContainerDied","Data":"1185c22aef62a9e191980f1a601e168d0fdc24a4a343062a83fa942bcf2205eb"} Jul 28 22:06:10.637534 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:10.637140 2557 scope.go:117] "RemoveContainer" containerID="20d60ff2c724abebe6a8210a89628b0de2a5cc7bac006f6428b2721e3463feed" Jul 28 22:06:10.637534 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:10.637093 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776d948f57-5692h" Jul 28 22:06:10.653066 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:10.653043 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776d948f57-5692h"] Jul 28 22:06:10.656738 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:10.656717 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-776d948f57-5692h"] Jul 28 22:06:12.045131 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:12.045098 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16902eed-a0c9-4180-8b79-3cb0795cd6d9" path="/var/lib/kubelet/pods/16902eed-a0c9-4180-8b79-3cb0795cd6d9/volumes" Jul 28 22:06:15.370338 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:15.370241 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:06:15.370338 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:15.370320 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:06:19.491622 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:19.491593 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g6fnc" Jul 28 22:06:33.139314 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.139252 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bf8854547-qcp2f" podUID="aacdc7b8-0c5e-44a2-b600-0e9b903ff402" containerName="console" containerID="cri-o://fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783" gracePeriod=15 Jul 28 22:06:33.397131 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.397065 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:06:33.571118 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.570680 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-trusted-ca-bundle\") pod \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " Jul 28 22:06:33.571118 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.570770 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-oauth-serving-cert\") pod \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " Jul 28 22:06:33.571351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571136 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aacdc7b8-0c5e-44a2-b600-0e9b903ff402" (UID: "aacdc7b8-0c5e-44a2-b600-0e9b903ff402"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:06:33.571351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571174 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aacdc7b8-0c5e-44a2-b600-0e9b903ff402" (UID: "aacdc7b8-0c5e-44a2-b600-0e9b903ff402"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:06:33.571351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571192 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-service-ca\") pod \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " Jul 28 22:06:33.571351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571232 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-oauth-config\") pod \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " Jul 28 22:06:33.571351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571257 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twhvm\" (UniqueName: \"kubernetes.io/projected/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-kube-api-access-twhvm\") pod \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " Jul 28 22:06:33.571351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571290 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-serving-cert\") pod \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " Jul 28 22:06:33.571351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571322 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-config\") pod \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\" (UID: \"aacdc7b8-0c5e-44a2-b600-0e9b903ff402\") " Jul 28 22:06:33.571708 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571450 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-service-ca" (OuterVolumeSpecName: "service-ca") pod "aacdc7b8-0c5e-44a2-b600-0e9b903ff402" (UID: "aacdc7b8-0c5e-44a2-b600-0e9b903ff402"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:06:33.571708 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571630 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-oauth-serving-cert\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:33.571708 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571649 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-service-ca\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:33.571708 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571664 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-trusted-ca-bundle\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:33.571908 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.571708 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-config" (OuterVolumeSpecName: "console-config") pod "aacdc7b8-0c5e-44a2-b600-0e9b903ff402" (UID: "aacdc7b8-0c5e-44a2-b600-0e9b903ff402"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:06:33.573610 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.573583 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aacdc7b8-0c5e-44a2-b600-0e9b903ff402" (UID: "aacdc7b8-0c5e-44a2-b600-0e9b903ff402"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:06:33.573722 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.573680 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aacdc7b8-0c5e-44a2-b600-0e9b903ff402" (UID: "aacdc7b8-0c5e-44a2-b600-0e9b903ff402"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:06:33.573762 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.573712 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-kube-api-access-twhvm" (OuterVolumeSpecName: "kube-api-access-twhvm") pod "aacdc7b8-0c5e-44a2-b600-0e9b903ff402" (UID: "aacdc7b8-0c5e-44a2-b600-0e9b903ff402"). InnerVolumeSpecName "kube-api-access-twhvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:06:33.672973 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.672858 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-oauth-config\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:33.672973 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.672891 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twhvm\" (UniqueName: \"kubernetes.io/projected/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-kube-api-access-twhvm\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:33.672973 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.672900 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-serving-cert\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:33.672973 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.672909 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aacdc7b8-0c5e-44a2-b600-0e9b903ff402-console-config\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:06:33.739601 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.739570 2557 generic.go:356] "Generic (PLEG): container finished" podID="aacdc7b8-0c5e-44a2-b600-0e9b903ff402" containerID="fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783" exitCode=0 Jul 28 22:06:33.739787 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.739618 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf8854547-qcp2f" Jul 28 22:06:33.739787 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.739627 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf8854547-qcp2f" event={"ID":"aacdc7b8-0c5e-44a2-b600-0e9b903ff402","Type":"ContainerDied","Data":"fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783"} Jul 28 22:06:33.739787 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.739676 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf8854547-qcp2f" event={"ID":"aacdc7b8-0c5e-44a2-b600-0e9b903ff402","Type":"ContainerDied","Data":"55887fdd8eea4980e98d79c82931b87b107d51d014edaaa3c89bc04980cd8b79"} Jul 28 22:06:33.739787 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.739702 2557 scope.go:117] "RemoveContainer" containerID="fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783" Jul 28 22:06:33.748561 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.748540 2557 scope.go:117] "RemoveContainer" containerID="fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783" Jul 28 22:06:33.748848 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:06:33.748829 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783\": container with ID starting with fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783 not found: ID does not exist" containerID="fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783" Jul 28 22:06:33.748900 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.748856 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783"} err="failed to get container status \"fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783\": rpc error: code = NotFound desc = could not find container \"fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783\": container with ID starting with fa14f200644f4434ea3e9068c21ecf2c2a7330c634524026f49a183cfe069783 not found: ID does not exist" Jul 28 22:06:33.766557 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.766529 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf8854547-qcp2f"] Jul 28 22:06:33.777602 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:33.777573 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bf8854547-qcp2f"] Jul 28 22:06:34.044560 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:34.044528 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aacdc7b8-0c5e-44a2-b600-0e9b903ff402" path="/var/lib/kubelet/pods/aacdc7b8-0c5e-44a2-b600-0e9b903ff402/volumes" Jul 28 22:06:35.375598 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:35.375561 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:06:35.379255 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:35.379232 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bc7996b9-g9q79" Jul 28 22:06:57.241959 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:57.241912 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:06:57.261521 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:57.261496 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:06:57.851353 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:06:57.851325 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 22:09:04.059237 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.059123 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hbqm6"] Jul 28 22:09:04.059827 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.059595 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aacdc7b8-0c5e-44a2-b600-0e9b903ff402" containerName="console" Jul 28 22:09:04.059827 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.059613 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aacdc7b8-0c5e-44a2-b600-0e9b903ff402" containerName="console" Jul 28 22:09:04.059827 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.059635 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16902eed-a0c9-4180-8b79-3cb0795cd6d9" containerName="console" Jul 28 22:09:04.059827 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.059643 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="16902eed-a0c9-4180-8b79-3cb0795cd6d9" containerName="console" Jul 28 22:09:04.059827 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.059729 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="16902eed-a0c9-4180-8b79-3cb0795cd6d9" containerName="console" Jul 28 22:09:04.059827 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.059744 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="aacdc7b8-0c5e-44a2-b600-0e9b903ff402" containerName="console" Jul 28 22:09:04.061828 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.061809 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:04.065488 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.065465 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 22:09:04.066358 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.066340 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 22:09:04.066464 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.066343 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8dg5k\"" Jul 28 22:09:04.068803 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.068780 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fht8\" (UniqueName: \"kubernetes.io/projected/280ccb3e-dd19-4c6d-86b8-79312cd77d17-kube-api-access-6fht8\") pod \"cert-manager-webhook-6486ffd997-hbqm6\" (UID: \"280ccb3e-dd19-4c6d-86b8-79312cd77d17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:04.068891 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.068814 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/280ccb3e-dd19-4c6d-86b8-79312cd77d17-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hbqm6\" (UID: \"280ccb3e-dd19-4c6d-86b8-79312cd77d17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:04.094585 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.094556 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hbqm6"] Jul 28 22:09:04.169911 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.169875 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fht8\" (UniqueName: \"kubernetes.io/projected/280ccb3e-dd19-4c6d-86b8-79312cd77d17-kube-api-access-6fht8\") pod \"cert-manager-webhook-6486ffd997-hbqm6\" (UID: \"280ccb3e-dd19-4c6d-86b8-79312cd77d17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:04.170129 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.169953 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/280ccb3e-dd19-4c6d-86b8-79312cd77d17-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hbqm6\" (UID: \"280ccb3e-dd19-4c6d-86b8-79312cd77d17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:04.182040 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.182009 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fht8\" (UniqueName: \"kubernetes.io/projected/280ccb3e-dd19-4c6d-86b8-79312cd77d17-kube-api-access-6fht8\") pod \"cert-manager-webhook-6486ffd997-hbqm6\" (UID: \"280ccb3e-dd19-4c6d-86b8-79312cd77d17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:04.183710 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.183680 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/280ccb3e-dd19-4c6d-86b8-79312cd77d17-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hbqm6\" (UID: \"280ccb3e-dd19-4c6d-86b8-79312cd77d17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:04.378480 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.378386 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:04.503874 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:04.503849 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hbqm6"] Jul 28 22:09:04.505859 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:09:04.505834 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod280ccb3e_dd19_4c6d_86b8_79312cd77d17.slice/crio-5506b45a7b06e4650e53e15f6b375dc8adf9b72304038801ef5a68c023b69a1e WatchSource:0}: Error finding container 5506b45a7b06e4650e53e15f6b375dc8adf9b72304038801ef5a68c023b69a1e: Status 404 returned error can't find the container with id 5506b45a7b06e4650e53e15f6b375dc8adf9b72304038801ef5a68c023b69a1e Jul 28 22:09:05.323189 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:05.323159 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" event={"ID":"280ccb3e-dd19-4c6d-86b8-79312cd77d17","Type":"ContainerStarted","Data":"5506b45a7b06e4650e53e15f6b375dc8adf9b72304038801ef5a68c023b69a1e"} Jul 28 22:09:06.276524 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.276485 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2"] Jul 28 22:09:06.279572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.279541 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" Jul 28 22:09:06.282459 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.282310 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sl9nx\"" Jul 28 22:09:06.289700 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.289598 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2"] Jul 28 22:09:06.388014 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.387961 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k28d\" (UniqueName: \"kubernetes.io/projected/bde20143-fcbf-4746-b42a-70d184441a73-kube-api-access-2k28d\") pod \"cert-manager-cainjector-559cdd8c87-zhrt2\" (UID: \"bde20143-fcbf-4746-b42a-70d184441a73\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" Jul 28 22:09:06.388477 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.388097 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bde20143-fcbf-4746-b42a-70d184441a73-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zhrt2\" (UID: \"bde20143-fcbf-4746-b42a-70d184441a73\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" Jul 28 22:09:06.488869 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.488827 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bde20143-fcbf-4746-b42a-70d184441a73-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zhrt2\" (UID: \"bde20143-fcbf-4746-b42a-70d184441a73\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" Jul 28 22:09:06.489073 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.488914 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k28d\" (UniqueName: \"kubernetes.io/projected/bde20143-fcbf-4746-b42a-70d184441a73-kube-api-access-2k28d\") pod \"cert-manager-cainjector-559cdd8c87-zhrt2\" (UID: \"bde20143-fcbf-4746-b42a-70d184441a73\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" Jul 28 22:09:06.498058 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.498006 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bde20143-fcbf-4746-b42a-70d184441a73-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zhrt2\" (UID: \"bde20143-fcbf-4746-b42a-70d184441a73\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" Jul 28 22:09:06.498462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.498432 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k28d\" (UniqueName: \"kubernetes.io/projected/bde20143-fcbf-4746-b42a-70d184441a73-kube-api-access-2k28d\") pod \"cert-manager-cainjector-559cdd8c87-zhrt2\" (UID: \"bde20143-fcbf-4746-b42a-70d184441a73\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" Jul 28 22:09:06.593537 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:06.593456 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" Jul 28 22:09:08.072867 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:08.072837 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2"] Jul 28 22:09:08.075904 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:09:08.075875 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbde20143_fcbf_4746_b42a_70d184441a73.slice/crio-02a839457be053d9d19d78fd7eb40a3a65b5bdb8d1f0f0e6864d921215017927 WatchSource:0}: Error finding container 02a839457be053d9d19d78fd7eb40a3a65b5bdb8d1f0f0e6864d921215017927: Status 404 returned error can't find the container with id 02a839457be053d9d19d78fd7eb40a3a65b5bdb8d1f0f0e6864d921215017927 Jul 28 22:09:08.339032 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:08.339001 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" event={"ID":"280ccb3e-dd19-4c6d-86b8-79312cd77d17","Type":"ContainerStarted","Data":"f3f7296fb49957eb5fdc9fe2a88efce47be454450b6a156de7cf3ff835fb6317"} Jul 28 22:09:08.339236 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:08.339079 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:08.340609 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:08.340586 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" event={"ID":"bde20143-fcbf-4746-b42a-70d184441a73","Type":"ContainerStarted","Data":"653f84bc32fecb77fb6202d8629ffcda77e1ee693c6db324df9c90df53125914"} Jul 28 22:09:08.340719 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:08.340617 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" event={"ID":"bde20143-fcbf-4746-b42a-70d184441a73","Type":"ContainerStarted","Data":"02a839457be053d9d19d78fd7eb40a3a65b5bdb8d1f0f0e6864d921215017927"} Jul 28 22:09:08.355621 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:08.355571 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" podStartSLOduration=0.900508131 podStartE2EDuration="4.355555642s" totalImagesPullingTime="3.455047511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:08.355039691 +0000 UTC m=+272.992898845" watchObservedRunningTime="2026-07-28 22:09:08.355555642 +0000 UTC m=+272.993414770" Jul 28 22:09:08.370534 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:08.370483 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhrt2" podStartSLOduration=2.370471306 podStartE2EDuration="2.370471306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:08.370013428 +0000 UTC m=+273.007872558" watchObservedRunningTime="2026-07-28 22:09:08.370471306 +0000 UTC m=+273.008330434" Jul 28 22:09:14.347050 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:14.347021 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hbqm6" Jul 28 22:09:16.667267 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.667227 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5rpmk"] Jul 28 22:09:16.669529 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.669512 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5rpmk" Jul 28 22:09:16.672085 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.672062 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gpw8k\"" Jul 28 22:09:16.677317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.677290 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5rpmk"] Jul 28 22:09:16.782255 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.782216 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbdff2a-0849-4108-b51b-ee9d3ed48d6e-bound-sa-token\") pod \"cert-manager-55b4d954d5-5rpmk\" (UID: \"8cbdff2a-0849-4108-b51b-ee9d3ed48d6e\") " pod="cert-manager/cert-manager-55b4d954d5-5rpmk" Jul 28 22:09:16.782434 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.782286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9w8t\" (UniqueName: \"kubernetes.io/projected/8cbdff2a-0849-4108-b51b-ee9d3ed48d6e-kube-api-access-q9w8t\") pod \"cert-manager-55b4d954d5-5rpmk\" (UID: \"8cbdff2a-0849-4108-b51b-ee9d3ed48d6e\") " pod="cert-manager/cert-manager-55b4d954d5-5rpmk" Jul 28 22:09:16.883555 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.883514 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbdff2a-0849-4108-b51b-ee9d3ed48d6e-bound-sa-token\") pod \"cert-manager-55b4d954d5-5rpmk\" (UID: \"8cbdff2a-0849-4108-b51b-ee9d3ed48d6e\") " pod="cert-manager/cert-manager-55b4d954d5-5rpmk" Jul 28 22:09:16.883668 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.883594 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9w8t\" (UniqueName: \"kubernetes.io/projected/8cbdff2a-0849-4108-b51b-ee9d3ed48d6e-kube-api-access-q9w8t\") pod \"cert-manager-55b4d954d5-5rpmk\" (UID: \"8cbdff2a-0849-4108-b51b-ee9d3ed48d6e\") " pod="cert-manager/cert-manager-55b4d954d5-5rpmk" Jul 28 22:09:16.897744 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.897709 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbdff2a-0849-4108-b51b-ee9d3ed48d6e-bound-sa-token\") pod \"cert-manager-55b4d954d5-5rpmk\" (UID: \"8cbdff2a-0849-4108-b51b-ee9d3ed48d6e\") " pod="cert-manager/cert-manager-55b4d954d5-5rpmk" Jul 28 22:09:16.897882 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.897769 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9w8t\" (UniqueName: \"kubernetes.io/projected/8cbdff2a-0849-4108-b51b-ee9d3ed48d6e-kube-api-access-q9w8t\") pod \"cert-manager-55b4d954d5-5rpmk\" (UID: \"8cbdff2a-0849-4108-b51b-ee9d3ed48d6e\") " pod="cert-manager/cert-manager-55b4d954d5-5rpmk" Jul 28 22:09:16.979830 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:16.979785 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5rpmk" Jul 28 22:09:17.116643 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:17.116620 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5rpmk"] Jul 28 22:09:17.118610 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:09:17.118580 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cbdff2a_0849_4108_b51b_ee9d3ed48d6e.slice/crio-91d7dabadb088987488a0cfbec624bcb3b229b269bef0c1587d1ea52fc065aa7 WatchSource:0}: Error finding container 91d7dabadb088987488a0cfbec624bcb3b229b269bef0c1587d1ea52fc065aa7: Status 404 returned error can't find the container with id 91d7dabadb088987488a0cfbec624bcb3b229b269bef0c1587d1ea52fc065aa7 Jul 28 22:09:17.378445 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:17.378354 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5rpmk" event={"ID":"8cbdff2a-0849-4108-b51b-ee9d3ed48d6e","Type":"ContainerStarted","Data":"7b18cefe366142c8be31d730e711db26ae61c50730b40db678f304e9dbdd9174"} Jul 28 22:09:17.378445 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:17.378404 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5rpmk" event={"ID":"8cbdff2a-0849-4108-b51b-ee9d3ed48d6e","Type":"ContainerStarted","Data":"91d7dabadb088987488a0cfbec624bcb3b229b269bef0c1587d1ea52fc065aa7"} Jul 28 22:09:17.397443 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:17.397387 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5rpmk" podStartSLOduration=1.397369611 podStartE2EDuration="1.397369611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:17.395795726 +0000 UTC m=+282.033654857" watchObservedRunningTime="2026-07-28 22:09:17.397369611 +0000 UTC m=+282.035228740" Jul 28 22:09:35.597752 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.597708 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd"] Jul 28 22:09:35.604352 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.604329 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.607648 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.607623 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 22:09:35.607648 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.607642 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 22:09:35.608596 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.608577 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gbw8q\"" Jul 28 22:09:35.609117 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.609100 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 22:09:35.609167 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.609127 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 22:09:35.622537 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.622513 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd"] Jul 28 22:09:35.756418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.756381 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.756590 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.756450 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62hpb\" (UniqueName: \"kubernetes.io/projected/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-kube-api-access-62hpb\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.756590 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.756496 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-manifests\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.756590 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.756515 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-webhook-cert\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.756590 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.756567 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-tmp\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.794460 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.794432 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 22:09:35.857836 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.857756 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.857836 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.857813 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62hpb\" (UniqueName: \"kubernetes.io/projected/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-kube-api-access-62hpb\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.861507 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.857841 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-manifests\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.861507 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.858400 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-webhook-cert\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.861507 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.858489 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-tmp\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.861507 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.858700 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-manifests\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.861507 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.860939 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 22:09:35.861507 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.861423 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-tmp\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.866658 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.866638 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 22:09:35.870882 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.870854 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-webhook-cert\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.870882 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.870873 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.877432 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.877412 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 22:09:35.888426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.888400 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62hpb\" (UniqueName: \"kubernetes.io/projected/c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe-kube-api-access-62hpb\") pod \"opendatahub-operator-controller-manager-d45996775-v5tsd\" (UID: \"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe\") " pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:35.916349 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.916320 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gbw8q\"" Jul 28 22:09:35.923983 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:35.923952 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:36.063014 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:36.062992 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd"] Jul 28 22:09:36.065322 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:09:36.065287 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0acb1a8_d4ca_47a6_8d77_ec6dfa6bcdfe.slice/crio-22e3f5eab2e1a6b24944371ae9154a8e0ae26512d9e86e462e0470382d6a8795 WatchSource:0}: Error finding container 22e3f5eab2e1a6b24944371ae9154a8e0ae26512d9e86e462e0470382d6a8795: Status 404 returned error can't find the container with id 22e3f5eab2e1a6b24944371ae9154a8e0ae26512d9e86e462e0470382d6a8795 Jul 28 22:09:36.453202 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:36.453144 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" event={"ID":"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe","Type":"ContainerStarted","Data":"22e3f5eab2e1a6b24944371ae9154a8e0ae26512d9e86e462e0470382d6a8795"} Jul 28 22:09:39.469162 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:39.469127 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" event={"ID":"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe","Type":"ContainerStarted","Data":"9431296efe61bc724828a539fa5591998abcb192e5549bd4c78342a5c951ac14"} Jul 28 22:09:40.474766 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:40.474728 2557 generic.go:356] "Generic (PLEG): container finished" podID="c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe" containerID="9431296efe61bc724828a539fa5591998abcb192e5549bd4c78342a5c951ac14" exitCode=0 Jul 28 22:09:40.475174 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:40.474819 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" event={"ID":"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe","Type":"ContainerDied","Data":"9431296efe61bc724828a539fa5591998abcb192e5549bd4c78342a5c951ac14"} Jul 28 22:09:40.475358 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:40.475342 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:09:41.482131 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:41.482097 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" event={"ID":"c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe","Type":"ContainerStarted","Data":"3069ae00f2df333a31bcc08cd0d2c47869b2cdbbaba9fcb0568796a561c55abc"} Jul 28 22:09:41.482618 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:41.482197 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:41.508156 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:41.508101 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" podStartSLOduration=2.256365599 podStartE2EDuration="6.508084734s" totalImagesPullingTime="3.209337571s" totalInitContainerRuntime="1.042381564s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:41.50711915 +0000 UTC m=+306.144978280" watchObservedRunningTime="2026-07-28 22:09:41.508084734 +0000 UTC m=+306.145943863" Jul 28 22:09:52.488983 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:52.488918 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d45996775-v5tsd" Jul 28 22:09:54.446629 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.446591 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9"] Jul 28 22:09:54.455017 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.454989 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.459012 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.458984 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 22:09:54.459717 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.459694 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 22:09:54.459860 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.459740 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 22:09:54.460011 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.459994 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:09:54.460111 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.460074 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4wdp4\"" Jul 28 22:09:54.460281 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.460265 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 22:09:54.462344 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.462323 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9"] Jul 28 22:09:54.527560 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.527521 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c28ab4e-2288-4075-a2d4-603cc129f2f2-cert\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.527560 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.527561 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvz4f\" (UniqueName: \"kubernetes.io/projected/3c28ab4e-2288-4075-a2d4-603cc129f2f2-kube-api-access-mvz4f\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.527777 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.527594 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c28ab4e-2288-4075-a2d4-603cc129f2f2-metrics-cert\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.527777 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.527675 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c28ab4e-2288-4075-a2d4-603cc129f2f2-manager-config\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.629100 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.629066 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c28ab4e-2288-4075-a2d4-603cc129f2f2-manager-config\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.629308 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.629142 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c28ab4e-2288-4075-a2d4-603cc129f2f2-cert\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.629308 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.629171 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvz4f\" (UniqueName: \"kubernetes.io/projected/3c28ab4e-2288-4075-a2d4-603cc129f2f2-kube-api-access-mvz4f\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.629308 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.629207 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c28ab4e-2288-4075-a2d4-603cc129f2f2-metrics-cert\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.629824 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.629798 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c28ab4e-2288-4075-a2d4-603cc129f2f2-manager-config\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.631725 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.631704 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c28ab4e-2288-4075-a2d4-603cc129f2f2-cert\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.632120 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.632097 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c28ab4e-2288-4075-a2d4-603cc129f2f2-metrics-cert\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.649427 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.649400 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvz4f\" (UniqueName: \"kubernetes.io/projected/3c28ab4e-2288-4075-a2d4-603cc129f2f2-kube-api-access-mvz4f\") pod \"lws-controller-manager-64ddd775c9-5rjc9\" (UID: \"3c28ab4e-2288-4075-a2d4-603cc129f2f2\") " pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.766589 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.766553 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:54.914585 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:54.914562 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9"] Jul 28 22:09:54.916092 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:09:54.916064 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c28ab4e_2288_4075_a2d4_603cc129f2f2.slice/crio-fb29c3c6e6fede67d7e7c5f864f1eeac6d57ee416297bfe4f3dbacd5dcc4bb66 WatchSource:0}: Error finding container fb29c3c6e6fede67d7e7c5f864f1eeac6d57ee416297bfe4f3dbacd5dcc4bb66: Status 404 returned error can't find the container with id fb29c3c6e6fede67d7e7c5f864f1eeac6d57ee416297bfe4f3dbacd5dcc4bb66 Jul 28 22:09:55.548264 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:55.548226 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" event={"ID":"3c28ab4e-2288-4075-a2d4-603cc129f2f2","Type":"ContainerStarted","Data":"fb29c3c6e6fede67d7e7c5f864f1eeac6d57ee416297bfe4f3dbacd5dcc4bb66"} Jul 28 22:09:58.566697 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:58.566651 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" event={"ID":"3c28ab4e-2288-4075-a2d4-603cc129f2f2","Type":"ContainerStarted","Data":"6beca1f1839b957a24d8125b7eed8eb2efc3adbd6e4774ec2f0a30895446b839"} Jul 28 22:09:58.567127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:58.566748 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:09:58.601000 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:09:58.600944 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" podStartSLOduration=1.929899107 podStartE2EDuration="4.600909852s" totalImagesPullingTime="2.671010745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:58.599661761 +0000 UTC m=+323.237520888" watchObservedRunningTime="2026-07-28 22:09:58.600909852 +0000 UTC m=+323.238768981" Jul 28 22:10:03.022776 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.022741 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks"] Jul 28 22:10:03.036703 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.036672 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.039244 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.039214 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 22:10:03.039387 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.039254 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 22:10:03.039387 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.039285 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 22:10:03.039387 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.039316 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 22:10:03.039631 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.039611 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 22:10:03.039743 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.039726 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rfnr8\"" Jul 28 22:10:03.040584 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.040561 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 22:10:03.048346 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.048323 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks"] Jul 28 22:10:03.109488 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.109449 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78223ce8-8144-4324-b0d9-bbc11ae63ccc-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.109488 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.109489 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.109725 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.109522 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78223ce8-8144-4324-b0d9-bbc11ae63ccc-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.109725 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.109541 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.109725 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.109629 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.109725 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.109649 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr8zd\" (UniqueName: \"kubernetes.io/projected/78223ce8-8144-4324-b0d9-bbc11ae63ccc-kube-api-access-lr8zd\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.109725 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.109699 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.210746 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.210707 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78223ce8-8144-4324-b0d9-bbc11ae63ccc-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.210746 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.210752 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.211020 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.210813 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.211020 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.210841 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr8zd\" (UniqueName: \"kubernetes.io/projected/78223ce8-8144-4324-b0d9-bbc11ae63ccc-kube-api-access-lr8zd\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.211020 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.210869 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.211020 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.210994 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78223ce8-8144-4324-b0d9-bbc11ae63ccc-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.211020 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.211019 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.211498 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.211463 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.213112 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.213076 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78223ce8-8144-4324-b0d9-bbc11ae63ccc-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.213269 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.213249 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.213329 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.213269 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78223ce8-8144-4324-b0d9-bbc11ae63ccc-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.213329 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.213305 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.221432 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.221411 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78223ce8-8144-4324-b0d9-bbc11ae63ccc-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.223844 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.223824 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr8zd\" (UniqueName: \"kubernetes.io/projected/78223ce8-8144-4324-b0d9-bbc11ae63ccc-kube-api-access-lr8zd\") pod \"istiod-openshift-gateway-5677ccb49b-ck7ks\" (UID: \"78223ce8-8144-4324-b0d9-bbc11ae63ccc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.346954 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.346833 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:03.553150 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.553118 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks"] Jul 28 22:10:03.555169 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:10:03.555140 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78223ce8_8144_4324_b0d9_bbc11ae63ccc.slice/crio-2d67deff560bdbbeb1d994afab4eb51208316d454ab119e23909d2a466d41b23 WatchSource:0}: Error finding container 2d67deff560bdbbeb1d994afab4eb51208316d454ab119e23909d2a466d41b23: Status 404 returned error can't find the container with id 2d67deff560bdbbeb1d994afab4eb51208316d454ab119e23909d2a466d41b23 Jul 28 22:10:03.592039 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:03.591996 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" event={"ID":"78223ce8-8144-4324-b0d9-bbc11ae63ccc","Type":"ContainerStarted","Data":"2d67deff560bdbbeb1d994afab4eb51208316d454ab119e23909d2a466d41b23"} Jul 28 22:10:06.500946 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:06.500890 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 28 22:10:06.501205 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:06.500974 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 28 22:10:06.608770 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:06.608729 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" event={"ID":"78223ce8-8144-4324-b0d9-bbc11ae63ccc","Type":"ContainerStarted","Data":"be08cdf8e471cd84af6ab0095fb1763e69d195b0192d5e7472238a1c61f15bc2"} Jul 28 22:10:06.608979 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:06.608942 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:06.631343 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:06.631295 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" podStartSLOduration=1.688712821 podStartE2EDuration="4.631279731s" totalImagesPullingTime="2.94256691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:06.630321355 +0000 UTC m=+331.268180508" watchObservedRunningTime="2026-07-28 22:10:06.631279731 +0000 UTC m=+331.269138860" Jul 28 22:10:07.615675 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:07.615646 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ck7ks" Jul 28 22:10:09.573657 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:10:09.573625 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-64ddd775c9-5rjc9" Jul 28 22:11:50.273569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.273538 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx"] Jul 28 22:11:50.277180 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.277163 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:50.279826 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.279804 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5fr88\"" Jul 28 22:11:50.280307 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.280293 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 22:11:50.280361 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.280296 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 22:11:50.288302 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.288280 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx"] Jul 28 22:11:50.383700 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.383664 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzj7h\" (UniqueName: \"kubernetes.io/projected/9f250044-2b7d-433e-a58d-878fcb681bab-kube-api-access-kzj7h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" (UID: \"9f250044-2b7d-433e-a58d-878fcb681bab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:50.383888 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.383719 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f250044-2b7d-433e-a58d-878fcb681bab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" (UID: \"9f250044-2b7d-433e-a58d-878fcb681bab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:50.485001 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.484917 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f250044-2b7d-433e-a58d-878fcb681bab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" (UID: \"9f250044-2b7d-433e-a58d-878fcb681bab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:50.485167 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.485052 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzj7h\" (UniqueName: \"kubernetes.io/projected/9f250044-2b7d-433e-a58d-878fcb681bab-kube-api-access-kzj7h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" (UID: \"9f250044-2b7d-433e-a58d-878fcb681bab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:50.485363 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.485338 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f250044-2b7d-433e-a58d-878fcb681bab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" (UID: \"9f250044-2b7d-433e-a58d-878fcb681bab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:50.498674 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.498648 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzj7h\" (UniqueName: \"kubernetes.io/projected/9f250044-2b7d-433e-a58d-878fcb681bab-kube-api-access-kzj7h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" (UID: \"9f250044-2b7d-433e-a58d-878fcb681bab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:50.587802 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.587717 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:50.730816 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.730786 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx"] Jul 28 22:11:50.732809 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:11:50.732783 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f250044_2b7d_433e_a58d_878fcb681bab.slice/crio-03b747b6271caa88916bc860a28c8774e11d9c93f651cf12cb0d2d1e74d045eb WatchSource:0}: Error finding container 03b747b6271caa88916bc860a28c8774e11d9c93f651cf12cb0d2d1e74d045eb: Status 404 returned error can't find the container with id 03b747b6271caa88916bc860a28c8774e11d9c93f651cf12cb0d2d1e74d045eb Jul 28 22:11:50.931446 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.931360 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx"] Jul 28 22:11:50.933646 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.933622 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx"] Jul 28 22:11:50.950915 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.950892 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb"] Jul 28 22:11:50.955719 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.955698 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:11:50.966541 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:50.966518 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb"] Jul 28 22:11:51.090781 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:51.090754 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aa574f1-9e39-4902-b96e-1310e6edc87b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-s65jb\" (UID: \"0aa574f1-9e39-4902-b96e-1310e6edc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:11:51.090988 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:51.090802 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79px6\" (UniqueName: \"kubernetes.io/projected/0aa574f1-9e39-4902-b96e-1310e6edc87b-kube-api-access-79px6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-s65jb\" (UID: \"0aa574f1-9e39-4902-b96e-1310e6edc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:11:51.192050 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:51.191950 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aa574f1-9e39-4902-b96e-1310e6edc87b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-s65jb\" (UID: \"0aa574f1-9e39-4902-b96e-1310e6edc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:11:51.192050 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:51.192041 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79px6\" (UniqueName: \"kubernetes.io/projected/0aa574f1-9e39-4902-b96e-1310e6edc87b-kube-api-access-79px6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-s65jb\" (UID: \"0aa574f1-9e39-4902-b96e-1310e6edc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:11:51.192352 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:51.192329 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aa574f1-9e39-4902-b96e-1310e6edc87b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-s65jb\" (UID: \"0aa574f1-9e39-4902-b96e-1310e6edc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:11:51.203442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:51.203416 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79px6\" (UniqueName: \"kubernetes.io/projected/0aa574f1-9e39-4902-b96e-1310e6edc87b-kube-api-access-79px6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-s65jb\" (UID: \"0aa574f1-9e39-4902-b96e-1310e6edc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:11:51.267471 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:51.267438 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:11:51.392997 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:51.392972 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb"] Jul 28 22:11:51.395251 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:11:51.395230 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa574f1_9e39_4902_b96e_1310e6edc87b.slice/crio-2986f4fb7a0d487a59e5318fad6be8333ebf2db142b036a691b9940e4041d538 WatchSource:0}: Error finding container 2986f4fb7a0d487a59e5318fad6be8333ebf2db142b036a691b9940e4041d538: Status 404 returned error can't find the container with id 2986f4fb7a0d487a59e5318fad6be8333ebf2db142b036a691b9940e4041d538 Jul 28 22:11:52.084522 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:52.084492 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" event={"ID":"0aa574f1-9e39-4902-b96e-1310e6edc87b","Type":"ContainerStarted","Data":"2986f4fb7a0d487a59e5318fad6be8333ebf2db142b036a691b9940e4041d538"} Jul 28 22:11:56.113501 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.113464 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" event={"ID":"0aa574f1-9e39-4902-b96e-1310e6edc87b","Type":"ContainerStarted","Data":"f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3"} Jul 28 22:11:56.114093 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.113552 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:11:56.115341 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.115278 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" podUID="9f250044-2b7d-433e-a58d-878fcb681bab" containerName="manager" containerID="cri-o://59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027" gracePeriod=2 Jul 28 22:11:56.137715 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.137663 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" podStartSLOduration=1.9870761080000001 podStartE2EDuration="6.137646972s" totalImagesPullingTime="4.150570864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:56.13588187 +0000 UTC m=+440.773741004" watchObservedRunningTime="2026-07-28 22:11:56.137646972 +0000 UTC m=+440.775506101" Jul 28 22:11:56.138281 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:11:56.138241 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" is forbidden: User \"system:node:ip-10-0-142-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-225.ec2.internal' and this object" podUID="9f250044-2b7d-433e-a58d-878fcb681bab" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:56.359660 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.359635 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:56.362472 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:11:56.362439 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" is forbidden: User \"system:node:ip-10-0-142-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-225.ec2.internal' and this object" podUID="9f250044-2b7d-433e-a58d-878fcb681bab" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:56.447156 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.447065 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f250044-2b7d-433e-a58d-878fcb681bab-extensions-socket-volume\") pod \"9f250044-2b7d-433e-a58d-878fcb681bab\" (UID: \"9f250044-2b7d-433e-a58d-878fcb681bab\") " Jul 28 22:11:56.447304 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.447182 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzj7h\" (UniqueName: \"kubernetes.io/projected/9f250044-2b7d-433e-a58d-878fcb681bab-kube-api-access-kzj7h\") pod \"9f250044-2b7d-433e-a58d-878fcb681bab\" (UID: \"9f250044-2b7d-433e-a58d-878fcb681bab\") " Jul 28 22:11:56.447363 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.447340 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f250044-2b7d-433e-a58d-878fcb681bab-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9f250044-2b7d-433e-a58d-878fcb681bab" (UID: "9f250044-2b7d-433e-a58d-878fcb681bab"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:11:56.449329 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.449308 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f250044-2b7d-433e-a58d-878fcb681bab-kube-api-access-kzj7h" (OuterVolumeSpecName: "kube-api-access-kzj7h") pod "9f250044-2b7d-433e-a58d-878fcb681bab" (UID: "9f250044-2b7d-433e-a58d-878fcb681bab"). InnerVolumeSpecName "kube-api-access-kzj7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:11:56.548541 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.548505 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzj7h\" (UniqueName: \"kubernetes.io/projected/9f250044-2b7d-433e-a58d-878fcb681bab-kube-api-access-kzj7h\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:11:56.548541 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:56.548539 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f250044-2b7d-433e-a58d-878fcb681bab-extensions-socket-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:11:57.120751 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:57.120711 2557 generic.go:356] "Generic (PLEG): container finished" podID="9f250044-2b7d-433e-a58d-878fcb681bab" containerID="59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027" exitCode=2 Jul 28 22:11:57.121220 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:57.120762 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:57.121220 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:57.120824 2557 scope.go:117] "RemoveContainer" containerID="59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027" Jul 28 22:11:57.123345 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:11:57.123322 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" is forbidden: User \"system:node:ip-10-0-142-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-225.ec2.internal' and this object" podUID="9f250044-2b7d-433e-a58d-878fcb681bab" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:57.130194 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:57.130174 2557 scope.go:117] "RemoveContainer" containerID="59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027" Jul 28 22:11:57.130451 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:11:57.130435 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027\": container with ID starting with 59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027 not found: ID does not exist" containerID="59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027" Jul 28 22:11:57.130494 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:57.130463 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027"} err="failed to get container status \"59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027\": rpc error: code = NotFound desc = could not find container \"59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027\": container with ID starting with 59a2c67175d82916008d6e9a5d99eb2eb325592dc6a181458638fe5a63bd9027 not found: ID does not exist" Jul 28 22:11:57.130834 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:11:57.130808 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8rwjx\" is forbidden: User \"system:node:ip-10-0-142-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-225.ec2.internal' and this object" podUID="9f250044-2b7d-433e-a58d-878fcb681bab" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8rwjx" Jul 28 22:11:58.045773 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:11:58.045739 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f250044-2b7d-433e-a58d-878fcb681bab" path="/var/lib/kubelet/pods/9f250044-2b7d-433e-a58d-878fcb681bab/volumes" Jul 28 22:12:07.123337 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:07.123308 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:12:19.381590 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:19.381552 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb"] Jul 28 22:12:19.382129 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:19.381828 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" podUID="0aa574f1-9e39-4902-b96e-1310e6edc87b" containerName="manager" containerID="cri-o://f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3" gracePeriod=10 Jul 28 22:12:19.640176 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:19.640116 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:12:19.761082 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:19.761045 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79px6\" (UniqueName: \"kubernetes.io/projected/0aa574f1-9e39-4902-b96e-1310e6edc87b-kube-api-access-79px6\") pod \"0aa574f1-9e39-4902-b96e-1310e6edc87b\" (UID: \"0aa574f1-9e39-4902-b96e-1310e6edc87b\") " Jul 28 22:12:19.761253 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:19.761091 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aa574f1-9e39-4902-b96e-1310e6edc87b-extensions-socket-volume\") pod \"0aa574f1-9e39-4902-b96e-1310e6edc87b\" (UID: \"0aa574f1-9e39-4902-b96e-1310e6edc87b\") " Jul 28 22:12:19.761598 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:19.761574 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa574f1-9e39-4902-b96e-1310e6edc87b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0aa574f1-9e39-4902-b96e-1310e6edc87b" (UID: "0aa574f1-9e39-4902-b96e-1310e6edc87b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:12:19.763169 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:19.763142 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa574f1-9e39-4902-b96e-1310e6edc87b-kube-api-access-79px6" (OuterVolumeSpecName: "kube-api-access-79px6") pod "0aa574f1-9e39-4902-b96e-1310e6edc87b" (UID: "0aa574f1-9e39-4902-b96e-1310e6edc87b"). InnerVolumeSpecName "kube-api-access-79px6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:12:19.862487 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:19.862444 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79px6\" (UniqueName: \"kubernetes.io/projected/0aa574f1-9e39-4902-b96e-1310e6edc87b-kube-api-access-79px6\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:12:19.862487 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:19.862475 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aa574f1-9e39-4902-b96e-1310e6edc87b-extensions-socket-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:12:20.228831 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:20.228799 2557 generic.go:356] "Generic (PLEG): container finished" podID="0aa574f1-9e39-4902-b96e-1310e6edc87b" containerID="f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3" exitCode=0 Jul 28 22:12:20.229039 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:20.228844 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" Jul 28 22:12:20.229039 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:20.228869 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" event={"ID":"0aa574f1-9e39-4902-b96e-1310e6edc87b","Type":"ContainerDied","Data":"f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3"} Jul 28 22:12:20.229039 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:20.228914 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb" event={"ID":"0aa574f1-9e39-4902-b96e-1310e6edc87b","Type":"ContainerDied","Data":"2986f4fb7a0d487a59e5318fad6be8333ebf2db142b036a691b9940e4041d538"} Jul 28 22:12:20.229039 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:20.228969 2557 scope.go:117] "RemoveContainer" containerID="f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3" Jul 28 22:12:20.237685 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:20.237669 2557 scope.go:117] "RemoveContainer" containerID="f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3" Jul 28 22:12:20.237959 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:12:20.237939 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3\": container with ID starting with f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3 not found: ID does not exist" containerID="f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3" Jul 28 22:12:20.238030 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:20.237966 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3"} err="failed to get container status \"f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3\": rpc error: code = NotFound desc = could not find container \"f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3\": container with ID starting with f37d9a4f93260b0ea96eda4d4b2dd6f6ee5aa3a47a2f4a14d6ba510eb4079ca3 not found: ID does not exist" Jul 28 22:12:20.248237 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:20.248210 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb"] Jul 28 22:12:20.251031 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:20.251009 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-s65jb"] Jul 28 22:12:22.045009 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:22.044979 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aa574f1-9e39-4902-b96e-1310e6edc87b" path="/var/lib/kubelet/pods/0aa574f1-9e39-4902-b96e-1310e6edc87b/volumes" Jul 28 22:12:37.564919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.564882 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:12:37.565316 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.565254 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aa574f1-9e39-4902-b96e-1310e6edc87b" containerName="manager" Jul 28 22:12:37.565316 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.565264 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aa574f1-9e39-4902-b96e-1310e6edc87b" containerName="manager" Jul 28 22:12:37.565316 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.565282 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f250044-2b7d-433e-a58d-878fcb681bab" containerName="manager" Jul 28 22:12:37.565316 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.565287 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f250044-2b7d-433e-a58d-878fcb681bab" containerName="manager" Jul 28 22:12:37.565438 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.565350 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f250044-2b7d-433e-a58d-878fcb681bab" containerName="manager" Jul 28 22:12:37.565438 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.565361 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aa574f1-9e39-4902-b96e-1310e6edc87b" containerName="manager" Jul 28 22:12:37.568149 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.568134 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:12:37.571743 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.571721 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 22:12:37.572213 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.572197 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h7fc5\"" Jul 28 22:12:37.573062 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.573043 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 22:12:37.573293 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.573273 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 22:12:37.581884 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.581861 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:12:37.619477 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.619437 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:12:37.725625 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.725594 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6c8adc46-ef29-49c7-9319-249f5723a69a-config-file\") pod \"limitador-limitador-54d6d988d5-sffm4\" (UID: \"6c8adc46-ef29-49c7-9319-249f5723a69a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:12:37.725780 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.725693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4f78\" (UniqueName: \"kubernetes.io/projected/6c8adc46-ef29-49c7-9319-249f5723a69a-kube-api-access-k4f78\") pod \"limitador-limitador-54d6d988d5-sffm4\" (UID: \"6c8adc46-ef29-49c7-9319-249f5723a69a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:12:37.826970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.826864 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4f78\" (UniqueName: \"kubernetes.io/projected/6c8adc46-ef29-49c7-9319-249f5723a69a-kube-api-access-k4f78\") pod \"limitador-limitador-54d6d988d5-sffm4\" (UID: \"6c8adc46-ef29-49c7-9319-249f5723a69a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:12:37.826970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.826912 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6c8adc46-ef29-49c7-9319-249f5723a69a-config-file\") pod \"limitador-limitador-54d6d988d5-sffm4\" (UID: \"6c8adc46-ef29-49c7-9319-249f5723a69a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:12:37.827560 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.827541 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6c8adc46-ef29-49c7-9319-249f5723a69a-config-file\") pod \"limitador-limitador-54d6d988d5-sffm4\" (UID: \"6c8adc46-ef29-49c7-9319-249f5723a69a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:12:37.836723 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.836698 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4f78\" (UniqueName: \"kubernetes.io/projected/6c8adc46-ef29-49c7-9319-249f5723a69a-kube-api-access-k4f78\") pod \"limitador-limitador-54d6d988d5-sffm4\" (UID: \"6c8adc46-ef29-49c7-9319-249f5723a69a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:12:37.878572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:37.878531 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:12:38.007501 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:38.007478 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:12:38.009858 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:12:38.009830 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8adc46_ef29_49c7_9319_249f5723a69a.slice/crio-0e49e66692b0da183a6f36fbf49bfacc75e5ee0bc92a6e6d9cfcfa712d623e02 WatchSource:0}: Error finding container 0e49e66692b0da183a6f36fbf49bfacc75e5ee0bc92a6e6d9cfcfa712d623e02: Status 404 returned error can't find the container with id 0e49e66692b0da183a6f36fbf49bfacc75e5ee0bc92a6e6d9cfcfa712d623e02 Jul 28 22:12:38.317976 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:38.317938 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" event={"ID":"6c8adc46-ef29-49c7-9319-249f5723a69a","Type":"ContainerStarted","Data":"0e49e66692b0da183a6f36fbf49bfacc75e5ee0bc92a6e6d9cfcfa712d623e02"} Jul 28 22:12:41.335485 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:41.335446 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" event={"ID":"6c8adc46-ef29-49c7-9319-249f5723a69a","Type":"ContainerStarted","Data":"302cef586895c53086179f2226c9fe8f2c98583e4f5614d4e5098c967b84ec8f"} Jul 28 22:12:41.335883 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:41.335539 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:12:41.352474 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:41.352424 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" podStartSLOduration=1.988562747 podStartE2EDuration="4.352406723s" totalImagesPullingTime="2.363843976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:12:41.350599384 +0000 UTC m=+485.988458504" watchObservedRunningTime="2026-07-28 22:12:41.352406723 +0000 UTC m=+485.990265909" Jul 28 22:12:52.340302 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:12:52.340272 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-sffm4" Jul 28 22:13:52.181697 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.181658 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-m7ch8"] Jul 28 22:13:52.184719 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.184702 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:13:52.187435 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.187403 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8ssmp\"" Jul 28 22:13:52.187550 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.187517 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 28 22:13:52.194310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.194287 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-m7ch8"] Jul 28 22:13:52.229119 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.229085 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:13:52.302733 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.302687 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f207a0bb-d507-481b-934d-04eb809019ec-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-m7ch8\" (UID: \"f207a0bb-d507-481b-934d-04eb809019ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:13:52.302921 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.302888 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrw2\" (UniqueName: \"kubernetes.io/projected/f207a0bb-d507-481b-934d-04eb809019ec-kube-api-access-zlrw2\") pod \"payload-processing-5cc6cd444b-m7ch8\" (UID: \"f207a0bb-d507-481b-934d-04eb809019ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:13:52.403412 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.403367 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrw2\" (UniqueName: \"kubernetes.io/projected/f207a0bb-d507-481b-934d-04eb809019ec-kube-api-access-zlrw2\") pod \"payload-processing-5cc6cd444b-m7ch8\" (UID: \"f207a0bb-d507-481b-934d-04eb809019ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:13:52.403584 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.403422 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f207a0bb-d507-481b-934d-04eb809019ec-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-m7ch8\" (UID: \"f207a0bb-d507-481b-934d-04eb809019ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:13:52.404133 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.404114 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f207a0bb-d507-481b-934d-04eb809019ec-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-m7ch8\" (UID: \"f207a0bb-d507-481b-934d-04eb809019ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:13:52.411654 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.411630 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrw2\" (UniqueName: \"kubernetes.io/projected/f207a0bb-d507-481b-934d-04eb809019ec-kube-api-access-zlrw2\") pod \"payload-processing-5cc6cd444b-m7ch8\" (UID: \"f207a0bb-d507-481b-934d-04eb809019ec\") " pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:13:52.496594 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.496560 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:13:52.831359 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:52.831339 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-m7ch8"] Jul 28 22:13:52.833197 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:13:52.833169 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf207a0bb_d507_481b_934d_04eb809019ec.slice/crio-f4d27ef2b1e6d990197f63f73345c0d0eeeb650e54a24a44883ac216a3676035 WatchSource:0}: Error finding container f4d27ef2b1e6d990197f63f73345c0d0eeeb650e54a24a44883ac216a3676035: Status 404 returned error can't find the container with id f4d27ef2b1e6d990197f63f73345c0d0eeeb650e54a24a44883ac216a3676035 Jul 28 22:13:53.665494 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:53.665454 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" event={"ID":"f207a0bb-d507-481b-934d-04eb809019ec","Type":"ContainerStarted","Data":"f4d27ef2b1e6d990197f63f73345c0d0eeeb650e54a24a44883ac216a3676035"} Jul 28 22:13:54.672044 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:54.672003 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" event={"ID":"f207a0bb-d507-481b-934d-04eb809019ec","Type":"ContainerStarted","Data":"d0c65db87f9920dd6bfe7943e6cf4cb0d370d6009f835c64e6c353fe57948b93"} Jul 28 22:13:54.672459 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:54.672106 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:13:54.689198 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:13:54.689146 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" podStartSLOduration=1.310276534 podStartE2EDuration="2.689128066s" totalImagesPullingTime="1.378851532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:13:54.687334936 +0000 UTC m=+559.325194068" watchObservedRunningTime="2026-07-28 22:13:54.689128066 +0000 UTC m=+559.326987195" Jul 28 22:14:05.679003 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:05.678972 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-m7ch8" Jul 28 22:14:13.470385 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:13.470348 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:14:20.185542 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.185507 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd"] Jul 28 22:14:20.189019 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.188997 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.191748 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.191725 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 22:14:20.192838 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.192820 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 22:14:20.192948 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.192823 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2wf7s\"" Jul 28 22:14:20.192948 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.192823 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 28 22:14:20.200212 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.200189 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd"] Jul 28 22:14:20.251587 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.251551 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.251587 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.251587 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.251796 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.251615 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.251796 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.251696 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.251796 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.251750 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.251899 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.251797 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgw46\" (UniqueName: \"kubernetes.io/projected/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-kube-api-access-sgw46\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.352900 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.352864 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.353096 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.352960 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgw46\" (UniqueName: \"kubernetes.io/projected/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-kube-api-access-sgw46\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.353096 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.353026 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.353096 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.353051 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.353096 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.353083 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.353291 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.353174 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.353494 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.353468 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.353580 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.353494 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.353580 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.353518 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.355338 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.355316 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.355441 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.355407 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.360764 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.360741 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgw46\" (UniqueName: \"kubernetes.io/projected/0c67f0f7-35a3-4f78-9399-1fe16b6d3e07-kube-api-access-sgw46\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6zptd\" (UID: \"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.501239 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.501194 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:20.627260 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.627234 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd"] Jul 28 22:14:20.629373 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:14:20.629347 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c67f0f7_35a3_4f78_9399_1fe16b6d3e07.slice/crio-fa07dadf5da34f725fac308f114518485a71946461aaa64d8ec72d361516d17a WatchSource:0}: Error finding container fa07dadf5da34f725fac308f114518485a71946461aaa64d8ec72d361516d17a: Status 404 returned error can't find the container with id fa07dadf5da34f725fac308f114518485a71946461aaa64d8ec72d361516d17a Jul 28 22:14:20.791037 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:20.790953 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" event={"ID":"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07","Type":"ContainerStarted","Data":"fa07dadf5da34f725fac308f114518485a71946461aaa64d8ec72d361516d17a"} Jul 28 22:14:21.090514 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:21.090432 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:14:21.797332 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:21.797296 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" event={"ID":"0c67f0f7-35a3-4f78-9399-1fe16b6d3e07","Type":"ContainerStarted","Data":"90cce1524b990af3d8c75b12f221a83880dc8a2557c72a07c2062246a8afec20"} Jul 28 22:14:21.813500 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:21.813452 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" podStartSLOduration=0.870994449 podStartE2EDuration="1.813436755s" totalImagesPullingTime="942.442306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:21.812278163 +0000 UTC m=+586.450137317" watchObservedRunningTime="2026-07-28 22:14:21.813436755 +0000 UTC m=+586.451295883" Jul 28 22:14:30.502116 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:30.502051 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:30.502683 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:30.502162 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:30.514893 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:30.514864 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:30.849649 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:30.849562 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6zptd" Jul 28 22:14:33.600907 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:33.600868 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:14:39.289123 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:39.289091 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:14:51.891572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:51.891536 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh"] Jul 28 22:14:51.895371 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:51.895350 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:51.898063 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:51.898041 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 28 22:14:51.904599 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:51.904574 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh"] Jul 28 22:14:52.040956 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.040892 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.041148 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.040960 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.041148 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.041001 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.041148 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.041022 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbkpq\" (UniqueName: \"kubernetes.io/projected/a8333182-2a14-4fcc-b680-f929793f117e-kube-api-access-nbkpq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.041148 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.041049 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8333182-2a14-4fcc-b680-f929793f117e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.041148 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.041129 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.142671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.142582 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.142671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.142618 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.142671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.142640 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.142671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.142658 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbkpq\" (UniqueName: \"kubernetes.io/projected/a8333182-2a14-4fcc-b680-f929793f117e-kube-api-access-nbkpq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.142671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.142675 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8333182-2a14-4fcc-b680-f929793f117e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.143114 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.142862 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.143114 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.143056 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.143220 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.143158 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.143220 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.143174 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.145296 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.145268 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8333182-2a14-4fcc-b680-f929793f117e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.145802 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.145781 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8333182-2a14-4fcc-b680-f929793f117e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.151494 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.151470 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbkpq\" (UniqueName: \"kubernetes.io/projected/a8333182-2a14-4fcc-b680-f929793f117e-kube-api-access-nbkpq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gvgh\" (UID: \"a8333182-2a14-4fcc-b680-f929793f117e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.206488 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.206447 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:14:52.338783 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.338762 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh"] Jul 28 22:14:52.340713 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:14:52.340688 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8333182_2a14_4fcc_b680_f929793f117e.slice/crio-ecb4c2c3cc423b80817cc24175d1ea2fa14ed34f3e10a70558900e1b9668587f WatchSource:0}: Error finding container ecb4c2c3cc423b80817cc24175d1ea2fa14ed34f3e10a70558900e1b9668587f: Status 404 returned error can't find the container with id ecb4c2c3cc423b80817cc24175d1ea2fa14ed34f3e10a70558900e1b9668587f Jul 28 22:14:52.342460 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.342443 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:14:52.946459 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.946363 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" event={"ID":"a8333182-2a14-4fcc-b680-f929793f117e","Type":"ContainerStarted","Data":"d0ff685bddad3d89c22d3b729ab0707b96d053a66866a9581144c2d0dd984d9f"} Jul 28 22:14:52.946459 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.946425 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" event={"ID":"a8333182-2a14-4fcc-b680-f929793f117e","Type":"ContainerStarted","Data":"ecb4c2c3cc423b80817cc24175d1ea2fa14ed34f3e10a70558900e1b9668587f"} Jul 28 22:14:52.968570 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:52.968518 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" podStartSLOduration=1.793925292 podStartE2EDuration="1.968502268s" totalImagesPullingTime="174.576976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:52.967016819 +0000 UTC m=+617.604875948" watchObservedRunningTime="2026-07-28 22:14:52.968502268 +0000 UTC m=+617.606361396" Jul 28 22:14:54.796334 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:14:54.796299 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:15:02.207401 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:15:02.207314 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:15:02.207401 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:15:02.207372 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:15:02.219690 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:15:02.219669 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:15:03.011655 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:15:03.011627 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gvgh" Jul 28 22:15:08.796700 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:15:08.796663 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:16:39.592904 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:16:39.592856 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:16:48.906940 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:16:48.906896 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:16:59.903919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:16:59.903880 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:18:44.295350 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:18:44.295312 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:18:56.290975 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:18:56.290938 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:19:32.288601 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:19:32.288521 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:19:41.990500 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:19:41.990461 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:19:52.090127 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:19:52.090092 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:19:59.903541 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:19:59.903500 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:20:11.312746 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:20:11.312707 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:20:21.198478 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:20:21.198441 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:20:32.293411 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:20:32.293374 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:20:42.289397 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:20:42.289361 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:21:04.193384 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:21:04.193300 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:21:16.497324 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:21:16.497286 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:21:39.707351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:21:39.707312 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:21:49.587223 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:21:49.587190 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:21:59.593053 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:21:59.593019 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:22:16.697267 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:22:16.697225 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:22:27.990676 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:22:27.990594 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:22:37.698523 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:22:37.698485 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:22:49.094021 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:22:49.093985 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:23:01.297550 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:23:01.297512 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:24:53.988440 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:24:53.988398 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:25:05.096410 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:25:05.096370 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:25:19.696636 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:25:19.696579 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:25:29.208041 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:25:29.207950 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:25:42.800678 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:25:42.800639 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:25:51.092537 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:25:51.092503 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:26:01.099472 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:26:01.099435 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:26:13.498208 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:26:13.498169 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:26:21.599402 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:26:21.599363 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:26:31.197972 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:26:31.197915 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:26:43.937981 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:26:43.937943 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:26:51.215187 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:26:51.215137 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:27:09.496465 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:27:09.496382 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:27:18.895770 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:27:18.895736 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:27:38.089138 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:27:38.089096 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:27:57.300505 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:27:57.300470 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:28:11.500609 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:28:11.500563 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:28:25.992324 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:28:25.992285 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:28:33.094521 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:28:33.094438 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:28:49.197641 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:28:49.197597 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:28:59.292199 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:28:59.292164 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:29:09.093911 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:29:09.093872 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:29:19.605349 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:29:19.605309 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:29:29.013977 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:29:29.013936 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:29:40.203636 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:29:40.203600 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:29:52.300589 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:29:52.300552 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:30:07.299206 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:30:07.299122 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:30:17.097890 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:30:17.097851 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:30:24.492026 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:30:24.491987 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:30:35.089556 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:30:35.089516 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:30:44.198018 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:30:44.197982 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:30:56.995703 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:30:56.990540 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:31:07.795726 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:31:07.795688 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:31:19.399749 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:31:19.399713 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:31:27.800249 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:31:27.800152 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:31:42.098721 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:31:42.098678 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:31:55.029325 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:31:55.029285 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:32:04.129584 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.129551 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm"] Jul 28 22:32:04.133025 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.133009 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:04.135644 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.135617 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-09e3e515-dockercfg-v4bjk\"" Jul 28 22:32:04.135768 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.135629 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-09e3e515\"" Jul 28 22:32:04.141580 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.141553 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm"] Jul 28 22:32:04.165636 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.165597 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft"] Jul 28 22:32:04.169429 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.169403 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:04.178601 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.178576 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft"] Jul 28 22:32:04.272493 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.272455 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnztt\" (UniqueName: \"kubernetes.io/projected/6e241e08-926e-4c2b-85ff-b157947375ef-kube-api-access-qnztt\") pod \"payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm\" (UID: \"6e241e08-926e-4c2b-85ff-b157947375ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:04.272657 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.272503 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e241e08-926e-4c2b-85ff-b157947375ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm\" (UID: \"6e241e08-926e-4c2b-85ff-b157947375ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:04.272701 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.272652 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw2wq\" (UniqueName: \"kubernetes.io/projected/97bf361e-55f5-43c7-82b4-db11a369f3c6-kube-api-access-pw2wq\") pod \"payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft\" (UID: \"97bf361e-55f5-43c7-82b4-db11a369f3c6\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:04.272740 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.272694 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97bf361e-55f5-43c7-82b4-db11a369f3c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft\" (UID: \"97bf361e-55f5-43c7-82b4-db11a369f3c6\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:04.374224 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.374183 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw2wq\" (UniqueName: \"kubernetes.io/projected/97bf361e-55f5-43c7-82b4-db11a369f3c6-kube-api-access-pw2wq\") pod \"payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft\" (UID: \"97bf361e-55f5-43c7-82b4-db11a369f3c6\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:04.374420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.374235 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97bf361e-55f5-43c7-82b4-db11a369f3c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft\" (UID: \"97bf361e-55f5-43c7-82b4-db11a369f3c6\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:04.374420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.374273 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnztt\" (UniqueName: \"kubernetes.io/projected/6e241e08-926e-4c2b-85ff-b157947375ef-kube-api-access-qnztt\") pod \"payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm\" (UID: \"6e241e08-926e-4c2b-85ff-b157947375ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:04.374420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.374316 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e241e08-926e-4c2b-85ff-b157947375ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm\" (UID: \"6e241e08-926e-4c2b-85ff-b157947375ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:04.374991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.374962 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97bf361e-55f5-43c7-82b4-db11a369f3c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft\" (UID: \"97bf361e-55f5-43c7-82b4-db11a369f3c6\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:04.375108 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.375075 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e241e08-926e-4c2b-85ff-b157947375ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm\" (UID: \"6e241e08-926e-4c2b-85ff-b157947375ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:04.382131 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.382068 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw2wq\" (UniqueName: \"kubernetes.io/projected/97bf361e-55f5-43c7-82b4-db11a369f3c6-kube-api-access-pw2wq\") pod \"payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft\" (UID: \"97bf361e-55f5-43c7-82b4-db11a369f3c6\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:04.382229 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.382142 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnztt\" (UniqueName: \"kubernetes.io/projected/6e241e08-926e-4c2b-85ff-b157947375ef-kube-api-access-qnztt\") pod \"payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm\" (UID: \"6e241e08-926e-4c2b-85ff-b157947375ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:04.444827 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.444790 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:04.480671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.480634 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:04.610189 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.610150 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm"] Jul 28 22:32:04.611585 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:32:04.611546 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e241e08_926e_4c2b_85ff_b157947375ef.slice/crio-3163e7ff9fbcf518520bb3c4842a7e6919949561c2156a6cc3c9998283026efe WatchSource:0}: Error finding container 3163e7ff9fbcf518520bb3c4842a7e6919949561c2156a6cc3c9998283026efe: Status 404 returned error can't find the container with id 3163e7ff9fbcf518520bb3c4842a7e6919949561c2156a6cc3c9998283026efe Jul 28 22:32:04.636781 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.636758 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft"] Jul 28 22:32:04.638633 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:32:04.638612 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bf361e_55f5_43c7_82b4_db11a369f3c6.slice/crio-0e66457c443350e172a9aabfdf0f9acfbf51a7fd9ab39927c0cb95e01f533140 WatchSource:0}: Error finding container 0e66457c443350e172a9aabfdf0f9acfbf51a7fd9ab39927c0cb95e01f533140: Status 404 returned error can't find the container with id 0e66457c443350e172a9aabfdf0f9acfbf51a7fd9ab39927c0cb95e01f533140 Jul 28 22:32:04.860862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.860812 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" event={"ID":"97bf361e-55f5-43c7-82b4-db11a369f3c6","Type":"ContainerStarted","Data":"76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2"} Jul 28 22:32:04.861082 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.860937 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" event={"ID":"97bf361e-55f5-43c7-82b4-db11a369f3c6","Type":"ContainerStarted","Data":"0e66457c443350e172a9aabfdf0f9acfbf51a7fd9ab39927c0cb95e01f533140"} Jul 28 22:32:04.861082 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.860974 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:04.862462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.862436 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" event={"ID":"6e241e08-926e-4c2b-85ff-b157947375ef","Type":"ContainerStarted","Data":"65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0"} Jul 28 22:32:04.862582 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.862483 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" event={"ID":"6e241e08-926e-4c2b-85ff-b157947375ef","Type":"ContainerStarted","Data":"3163e7ff9fbcf518520bb3c4842a7e6919949561c2156a6cc3c9998283026efe"} Jul 28 22:32:04.862582 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.862503 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:04.877898 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.877851 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" podStartSLOduration=0.877837431 podStartE2EDuration="877.837431ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:32:04.876012561 +0000 UTC m=+1649.513871693" watchObservedRunningTime="2026-07-28 22:32:04.877837431 +0000 UTC m=+1649.515696560" Jul 28 22:32:04.890653 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:04.890548 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" podStartSLOduration=0.890531928 podStartE2EDuration="890.531928ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:32:04.890147286 +0000 UTC m=+1649.528006416" watchObservedRunningTime="2026-07-28 22:32:04.890531928 +0000 UTC m=+1649.528391059" Jul 28 22:32:08.890802 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:08.890774 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:32:15.869545 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:15.869514 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:15.870050 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:15.869841 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:34.017093 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.017059 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft"] Jul 28 22:32:34.017583 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.017279 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" podUID="97bf361e-55f5-43c7-82b4-db11a369f3c6" containerName="payload-processing" containerID="cri-o://76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2" gracePeriod=30 Jul 28 22:32:34.025845 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.025813 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm"] Jul 28 22:32:34.026119 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.026090 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" podUID="6e241e08-926e-4c2b-85ff-b157947375ef" containerName="payload-pre-processing" containerID="cri-o://65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0" gracePeriod=30 Jul 28 22:32:34.317327 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.317294 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:34.321064 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.321041 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:34.357232 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.357200 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97bf361e-55f5-43c7-82b4-db11a369f3c6-plugins-config-volume\") pod \"97bf361e-55f5-43c7-82b4-db11a369f3c6\" (UID: \"97bf361e-55f5-43c7-82b4-db11a369f3c6\") " Jul 28 22:32:34.357396 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.357241 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnztt\" (UniqueName: \"kubernetes.io/projected/6e241e08-926e-4c2b-85ff-b157947375ef-kube-api-access-qnztt\") pod \"6e241e08-926e-4c2b-85ff-b157947375ef\" (UID: \"6e241e08-926e-4c2b-85ff-b157947375ef\") " Jul 28 22:32:34.357445 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.357433 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e241e08-926e-4c2b-85ff-b157947375ef-plugins-config-volume\") pod \"6e241e08-926e-4c2b-85ff-b157947375ef\" (UID: \"6e241e08-926e-4c2b-85ff-b157947375ef\") " Jul 28 22:32:34.357494 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.357474 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw2wq\" (UniqueName: \"kubernetes.io/projected/97bf361e-55f5-43c7-82b4-db11a369f3c6-kube-api-access-pw2wq\") pod \"97bf361e-55f5-43c7-82b4-db11a369f3c6\" (UID: \"97bf361e-55f5-43c7-82b4-db11a369f3c6\") " Jul 28 22:32:34.357656 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.357628 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bf361e-55f5-43c7-82b4-db11a369f3c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97bf361e-55f5-43c7-82b4-db11a369f3c6" (UID: "97bf361e-55f5-43c7-82b4-db11a369f3c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:32:34.357849 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.357828 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e241e08-926e-4c2b-85ff-b157947375ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e241e08-926e-4c2b-85ff-b157947375ef" (UID: "6e241e08-926e-4c2b-85ff-b157947375ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:32:34.357959 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.357863 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97bf361e-55f5-43c7-82b4-db11a369f3c6-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:32:34.359621 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.359599 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e241e08-926e-4c2b-85ff-b157947375ef-kube-api-access-qnztt" (OuterVolumeSpecName: "kube-api-access-qnztt") pod "6e241e08-926e-4c2b-85ff-b157947375ef" (UID: "6e241e08-926e-4c2b-85ff-b157947375ef"). InnerVolumeSpecName "kube-api-access-qnztt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:32:34.359721 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.359617 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bf361e-55f5-43c7-82b4-db11a369f3c6-kube-api-access-pw2wq" (OuterVolumeSpecName: "kube-api-access-pw2wq") pod "97bf361e-55f5-43c7-82b4-db11a369f3c6" (UID: "97bf361e-55f5-43c7-82b4-db11a369f3c6"). InnerVolumeSpecName "kube-api-access-pw2wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:32:34.459136 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.459100 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnztt\" (UniqueName: \"kubernetes.io/projected/6e241e08-926e-4c2b-85ff-b157947375ef-kube-api-access-qnztt\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:32:34.459136 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.459128 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e241e08-926e-4c2b-85ff-b157947375ef-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:32:34.459136 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:34.459137 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw2wq\" (UniqueName: \"kubernetes.io/projected/97bf361e-55f5-43c7-82b4-db11a369f3c6-kube-api-access-pw2wq\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:32:35.007767 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.007731 2557 generic.go:356] "Generic (PLEG): container finished" podID="6e241e08-926e-4c2b-85ff-b157947375ef" containerID="65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0" exitCode=0 Jul 28 22:32:35.007983 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.007783 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" Jul 28 22:32:35.007983 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.007827 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" event={"ID":"6e241e08-926e-4c2b-85ff-b157947375ef","Type":"ContainerDied","Data":"65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0"} Jul 28 22:32:35.007983 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.007889 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm" event={"ID":"6e241e08-926e-4c2b-85ff-b157947375ef","Type":"ContainerDied","Data":"3163e7ff9fbcf518520bb3c4842a7e6919949561c2156a6cc3c9998283026efe"} Jul 28 22:32:35.007983 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.007914 2557 scope.go:117] "RemoveContainer" containerID="65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0" Jul 28 22:32:35.009284 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.009262 2557 generic.go:356] "Generic (PLEG): container finished" podID="97bf361e-55f5-43c7-82b4-db11a369f3c6" containerID="76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2" exitCode=0 Jul 28 22:32:35.009446 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.009302 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" event={"ID":"97bf361e-55f5-43c7-82b4-db11a369f3c6","Type":"ContainerDied","Data":"76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2"} Jul 28 22:32:35.009446 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.009314 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" Jul 28 22:32:35.009446 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.009337 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft" event={"ID":"97bf361e-55f5-43c7-82b4-db11a369f3c6","Type":"ContainerDied","Data":"0e66457c443350e172a9aabfdf0f9acfbf51a7fd9ab39927c0cb95e01f533140"} Jul 28 22:32:35.017690 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.017667 2557 scope.go:117] "RemoveContainer" containerID="65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0" Jul 28 22:32:35.018166 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:32:35.018133 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0\": container with ID starting with 65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0 not found: ID does not exist" containerID="65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0" Jul 28 22:32:35.018285 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.018173 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0"} err="failed to get container status \"65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0\": rpc error: code = NotFound desc = could not find container \"65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0\": container with ID starting with 65f781920a1f3fce07895310a9a5c7073cd60035142d0e4af86f4ebdc59074d0 not found: ID does not exist" Jul 28 22:32:35.018285 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.018191 2557 scope.go:117] "RemoveContainer" containerID="76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2" Jul 28 22:32:35.028694 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.028671 2557 scope.go:117] "RemoveContainer" containerID="76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2" Jul 28 22:32:35.029039 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:32:35.029018 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2\": container with ID starting with 76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2 not found: ID does not exist" containerID="76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2" Jul 28 22:32:35.029156 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.029045 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2"} err="failed to get container status \"76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2\": rpc error: code = NotFound desc = could not find container \"76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2\": container with ID starting with 76b205b1581d35fdae5fdd46d969cf7b594b7b60f955191e5221d3a242c98eb2 not found: ID does not exist" Jul 28 22:32:35.031980 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.031954 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm"] Jul 28 22:32:35.039057 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.037603 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09e3e515-74cbfff6c6-t28qm"] Jul 28 22:32:35.052799 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.052768 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft"] Jul 28 22:32:35.058228 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:35.058200 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3e515-5f8ff6fb6-jqkft"] Jul 28 22:32:36.046318 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:36.046285 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e241e08-926e-4c2b-85ff-b157947375ef" path="/var/lib/kubelet/pods/6e241e08-926e-4c2b-85ff-b157947375ef/volumes" Jul 28 22:32:36.046684 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:36.046637 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97bf361e-55f5-43c7-82b4-db11a369f3c6" path="/var/lib/kubelet/pods/97bf361e-55f5-43c7-82b4-db11a369f3c6/volumes" Jul 28 22:32:38.504653 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.504615 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:32:38.521605 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.521569 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj"] Jul 28 22:32:38.521985 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.521972 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bf361e-55f5-43c7-82b4-db11a369f3c6" containerName="payload-processing" Jul 28 22:32:38.522033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.521987 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bf361e-55f5-43c7-82b4-db11a369f3c6" containerName="payload-processing" Jul 28 22:32:38.522033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.522002 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e241e08-926e-4c2b-85ff-b157947375ef" containerName="payload-pre-processing" Jul 28 22:32:38.522033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.522012 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e241e08-926e-4c2b-85ff-b157947375ef" containerName="payload-pre-processing" Jul 28 22:32:38.522132 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.522093 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bf361e-55f5-43c7-82b4-db11a369f3c6" containerName="payload-processing" Jul 28 22:32:38.522132 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.522105 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e241e08-926e-4c2b-85ff-b157947375ef" containerName="payload-pre-processing" Jul 28 22:32:38.526411 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.526388 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:38.529278 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.529252 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6a50116a-dockercfg-w4764\"" Jul 28 22:32:38.529511 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.529495 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6a50116a\"" Jul 28 22:32:38.534880 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.534699 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj"] Jul 28 22:32:38.538317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.538293 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw"] Jul 28 22:32:38.541826 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.541806 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:32:38.555322 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.555294 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw"] Jul 28 22:32:38.598089 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.598054 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ab430-750c-4140-bdbe-f0d8a6540637-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj\" (UID: \"ef1ab430-750c-4140-bdbe-f0d8a6540637\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:38.598287 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.598123 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc862\" (UniqueName: \"kubernetes.io/projected/ef1ab430-750c-4140-bdbe-f0d8a6540637-kube-api-access-wc862\") pod \"payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj\" (UID: \"ef1ab430-750c-4140-bdbe-f0d8a6540637\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:38.598335 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.598295 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa2eb36-b210-42c6-bf0c-20f710d3e622-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw\" (UID: \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:32:38.598378 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.598353 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmhzv\" (UniqueName: \"kubernetes.io/projected/dfa2eb36-b210-42c6-bf0c-20f710d3e622-kube-api-access-cmhzv\") pod \"payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw\" (UID: \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:32:38.699006 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.698954 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ab430-750c-4140-bdbe-f0d8a6540637-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj\" (UID: \"ef1ab430-750c-4140-bdbe-f0d8a6540637\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:38.699006 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.699009 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc862\" (UniqueName: \"kubernetes.io/projected/ef1ab430-750c-4140-bdbe-f0d8a6540637-kube-api-access-wc862\") pod \"payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj\" (UID: \"ef1ab430-750c-4140-bdbe-f0d8a6540637\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:38.699271 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.699047 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa2eb36-b210-42c6-bf0c-20f710d3e622-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw\" (UID: \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:32:38.699271 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.699076 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhzv\" (UniqueName: \"kubernetes.io/projected/dfa2eb36-b210-42c6-bf0c-20f710d3e622-kube-api-access-cmhzv\") pod \"payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw\" (UID: \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:32:38.699667 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.699645 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa2eb36-b210-42c6-bf0c-20f710d3e622-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw\" (UID: \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:32:38.699862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.699845 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ab430-750c-4140-bdbe-f0d8a6540637-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj\" (UID: \"ef1ab430-750c-4140-bdbe-f0d8a6540637\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:38.707620 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.707588 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhzv\" (UniqueName: \"kubernetes.io/projected/dfa2eb36-b210-42c6-bf0c-20f710d3e622-kube-api-access-cmhzv\") pod \"payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw\" (UID: \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:32:38.707620 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.707598 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc862\" (UniqueName: \"kubernetes.io/projected/ef1ab430-750c-4140-bdbe-f0d8a6540637-kube-api-access-wc862\") pod \"payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj\" (UID: \"ef1ab430-750c-4140-bdbe-f0d8a6540637\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:38.840349 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.840252 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:38.852369 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.852336 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:32:38.985184 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:38.985146 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj"] Jul 28 22:32:38.990225 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:32:38.990178 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1ab430_750c_4140_bdbe_f0d8a6540637.slice/crio-2ed0dd89a64bbee90499f5bec826f6429b0c0bd22d246abf7ca14cf920dfc2ba WatchSource:0}: Error finding container 2ed0dd89a64bbee90499f5bec826f6429b0c0bd22d246abf7ca14cf920dfc2ba: Status 404 returned error can't find the container with id 2ed0dd89a64bbee90499f5bec826f6429b0c0bd22d246abf7ca14cf920dfc2ba Jul 28 22:32:39.013310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:39.013284 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw"] Jul 28 22:32:39.016198 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:32:39.016173 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa2eb36_b210_42c6_bf0c_20f710d3e622.slice/crio-534898f616189561dccd60e012a4a29a7cdf4dd4d83abf35fae2a21dfa2e0eea WatchSource:0}: Error finding container 534898f616189561dccd60e012a4a29a7cdf4dd4d83abf35fae2a21dfa2e0eea: Status 404 returned error can't find the container with id 534898f616189561dccd60e012a4a29a7cdf4dd4d83abf35fae2a21dfa2e0eea Jul 28 22:32:39.031696 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:39.031660 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" event={"ID":"ef1ab430-750c-4140-bdbe-f0d8a6540637","Type":"ContainerStarted","Data":"2ed0dd89a64bbee90499f5bec826f6429b0c0bd22d246abf7ca14cf920dfc2ba"} Jul 28 22:32:39.032974 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:39.032951 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" event={"ID":"dfa2eb36-b210-42c6-bf0c-20f710d3e622","Type":"ContainerStarted","Data":"534898f616189561dccd60e012a4a29a7cdf4dd4d83abf35fae2a21dfa2e0eea"} Jul 28 22:32:40.048733 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:40.048704 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:40.049109 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:40.048755 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" event={"ID":"ef1ab430-750c-4140-bdbe-f0d8a6540637","Type":"ContainerStarted","Data":"b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed"} Jul 28 22:32:40.049109 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:40.048779 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" event={"ID":"dfa2eb36-b210-42c6-bf0c-20f710d3e622","Type":"ContainerStarted","Data":"8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7"} Jul 28 22:32:40.049109 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:40.048799 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:32:40.059081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:40.059033 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" podStartSLOduration=2.059017315 podStartE2EDuration="2.059017315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:32:40.057878179 +0000 UTC m=+1684.695737311" watchObservedRunningTime="2026-07-28 22:32:40.059017315 +0000 UTC m=+1684.696876421" Jul 28 22:32:40.072859 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:40.072804 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" podStartSLOduration=2.072787377 podStartE2EDuration="2.072787377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:32:40.07228108 +0000 UTC m=+1684.710140260" watchObservedRunningTime="2026-07-28 22:32:40.072787377 +0000 UTC m=+1684.710646505" Jul 28 22:32:50.295382 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:50.295343 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:32:51.049379 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:51.049349 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:32:51.049572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:32:51.049409 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:33:09.179542 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.179442 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw"] Jul 28 22:33:09.180442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.180345 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" podUID="dfa2eb36-b210-42c6-bf0c-20f710d3e622" containerName="payload-processing" containerID="cri-o://8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7" gracePeriod=30 Jul 28 22:33:09.193564 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.192513 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj"] Jul 28 22:33:09.193752 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.193723 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" podUID="ef1ab430-750c-4140-bdbe-f0d8a6540637" containerName="payload-pre-processing" containerID="cri-o://b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed" gracePeriod=30 Jul 28 22:33:09.480245 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.480218 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:33:09.483909 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.483876 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:33:09.492437 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.492412 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ab430-750c-4140-bdbe-f0d8a6540637-plugins-config-volume\") pod \"ef1ab430-750c-4140-bdbe-f0d8a6540637\" (UID: \"ef1ab430-750c-4140-bdbe-f0d8a6540637\") " Jul 28 22:33:09.492582 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.492447 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmhzv\" (UniqueName: \"kubernetes.io/projected/dfa2eb36-b210-42c6-bf0c-20f710d3e622-kube-api-access-cmhzv\") pod \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\" (UID: \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\") " Jul 28 22:33:09.492582 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.492478 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa2eb36-b210-42c6-bf0c-20f710d3e622-plugins-config-volume\") pod \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\" (UID: \"dfa2eb36-b210-42c6-bf0c-20f710d3e622\") " Jul 28 22:33:09.492582 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.492551 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc862\" (UniqueName: \"kubernetes.io/projected/ef1ab430-750c-4140-bdbe-f0d8a6540637-kube-api-access-wc862\") pod \"ef1ab430-750c-4140-bdbe-f0d8a6540637\" (UID: \"ef1ab430-750c-4140-bdbe-f0d8a6540637\") " Jul 28 22:33:09.492820 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.492798 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1ab430-750c-4140-bdbe-f0d8a6540637-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef1ab430-750c-4140-bdbe-f0d8a6540637" (UID: "ef1ab430-750c-4140-bdbe-f0d8a6540637"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:33:09.492937 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.492902 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa2eb36-b210-42c6-bf0c-20f710d3e622-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfa2eb36-b210-42c6-bf0c-20f710d3e622" (UID: "dfa2eb36-b210-42c6-bf0c-20f710d3e622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:33:09.494579 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.494553 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa2eb36-b210-42c6-bf0c-20f710d3e622-kube-api-access-cmhzv" (OuterVolumeSpecName: "kube-api-access-cmhzv") pod "dfa2eb36-b210-42c6-bf0c-20f710d3e622" (UID: "dfa2eb36-b210-42c6-bf0c-20f710d3e622"). InnerVolumeSpecName "kube-api-access-cmhzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:33:09.494633 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.494604 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1ab430-750c-4140-bdbe-f0d8a6540637-kube-api-access-wc862" (OuterVolumeSpecName: "kube-api-access-wc862") pod "ef1ab430-750c-4140-bdbe-f0d8a6540637" (UID: "ef1ab430-750c-4140-bdbe-f0d8a6540637"). InnerVolumeSpecName "kube-api-access-wc862". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:33:09.593448 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.593409 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc862\" (UniqueName: \"kubernetes.io/projected/ef1ab430-750c-4140-bdbe-f0d8a6540637-kube-api-access-wc862\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:33:09.593448 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.593436 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ab430-750c-4140-bdbe-f0d8a6540637-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:33:09.593448 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.593447 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmhzv\" (UniqueName: \"kubernetes.io/projected/dfa2eb36-b210-42c6-bf0c-20f710d3e622-kube-api-access-cmhzv\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:33:09.593698 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:09.593461 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa2eb36-b210-42c6-bf0c-20f710d3e622-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:33:10.202623 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.202582 2557 generic.go:356] "Generic (PLEG): container finished" podID="dfa2eb36-b210-42c6-bf0c-20f710d3e622" containerID="8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7" exitCode=0 Jul 28 22:33:10.203033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.202641 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" Jul 28 22:33:10.203033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.202792 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" event={"ID":"dfa2eb36-b210-42c6-bf0c-20f710d3e622","Type":"ContainerDied","Data":"8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7"} Jul 28 22:33:10.203033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.202846 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw" event={"ID":"dfa2eb36-b210-42c6-bf0c-20f710d3e622","Type":"ContainerDied","Data":"534898f616189561dccd60e012a4a29a7cdf4dd4d83abf35fae2a21dfa2e0eea"} Jul 28 22:33:10.203033 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.202879 2557 scope.go:117] "RemoveContainer" containerID="8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7" Jul 28 22:33:10.204737 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.204718 2557 generic.go:356] "Generic (PLEG): container finished" podID="ef1ab430-750c-4140-bdbe-f0d8a6540637" containerID="b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed" exitCode=0 Jul 28 22:33:10.204812 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.204754 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" event={"ID":"ef1ab430-750c-4140-bdbe-f0d8a6540637","Type":"ContainerDied","Data":"b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed"} Jul 28 22:33:10.204812 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.204760 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" Jul 28 22:33:10.204812 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.204797 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj" event={"ID":"ef1ab430-750c-4140-bdbe-f0d8a6540637","Type":"ContainerDied","Data":"2ed0dd89a64bbee90499f5bec826f6429b0c0bd22d246abf7ca14cf920dfc2ba"} Jul 28 22:33:10.212034 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.212015 2557 scope.go:117] "RemoveContainer" containerID="8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7" Jul 28 22:33:10.212622 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:33:10.212380 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7\": container with ID starting with 8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7 not found: ID does not exist" containerID="8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7" Jul 28 22:33:10.212622 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.212417 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7"} err="failed to get container status \"8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7\": rpc error: code = NotFound desc = could not find container \"8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7\": container with ID starting with 8ccf600e1de5b099fb56bdc714322bd2ff4f94058e8101b613c925ceee7855a7 not found: ID does not exist" Jul 28 22:33:10.212622 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.212441 2557 scope.go:117] "RemoveContainer" containerID="b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed" Jul 28 22:33:10.223335 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.223300 2557 scope.go:117] "RemoveContainer" containerID="b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed" Jul 28 22:33:10.224456 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:33:10.224313 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed\": container with ID starting with b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed not found: ID does not exist" containerID="b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed" Jul 28 22:33:10.224456 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.224353 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed"} err="failed to get container status \"b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed\": rpc error: code = NotFound desc = could not find container \"b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed\": container with ID starting with b704013a1c802e3e8f66bbbca3b081134d99a743cd7fac8db3467becdd7139ed not found: ID does not exist" Jul 28 22:33:10.230142 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.230107 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw"] Jul 28 22:33:10.250250 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.250222 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a50116a-db6764c85-w8ppw"] Jul 28 22:33:10.266803 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.266772 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj"] Jul 28 22:33:10.270875 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:10.270846 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a50116a-77c64657d8-l5fkj"] Jul 28 22:33:12.045719 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:12.045688 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa2eb36-b210-42c6-bf0c-20f710d3e622" path="/var/lib/kubelet/pods/dfa2eb36-b210-42c6-bf0c-20f710d3e622/volumes" Jul 28 22:33:12.046115 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:12.046048 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef1ab430-750c-4140-bdbe-f0d8a6540637" path="/var/lib/kubelet/pods/ef1ab430-750c-4140-bdbe-f0d8a6540637/volumes" Jul 28 22:33:13.604048 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.604018 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:33:13.608707 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.608680 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd"] Jul 28 22:33:13.609060 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.609046 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa2eb36-b210-42c6-bf0c-20f710d3e622" containerName="payload-processing" Jul 28 22:33:13.609117 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.609061 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa2eb36-b210-42c6-bf0c-20f710d3e622" containerName="payload-processing" Jul 28 22:33:13.609117 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.609090 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1ab430-750c-4140-bdbe-f0d8a6540637" containerName="payload-pre-processing" Jul 28 22:33:13.609117 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.609096 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1ab430-750c-4140-bdbe-f0d8a6540637" containerName="payload-pre-processing" Jul 28 22:33:13.609221 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.609145 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef1ab430-750c-4140-bdbe-f0d8a6540637" containerName="payload-pre-processing" Jul 28 22:33:13.609221 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.609158 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa2eb36-b210-42c6-bf0c-20f710d3e622" containerName="payload-processing" Jul 28 22:33:13.613801 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.613784 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:13.616689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.616667 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e1a8249b-dockercfg-mvfqb\"" Jul 28 22:33:13.616810 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.616668 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e1a8249b\"" Jul 28 22:33:13.624045 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.624018 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd"] Jul 28 22:33:13.627007 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.626982 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5qf\" (UniqueName: \"kubernetes.io/projected/d59fe046-6c9f-450a-b1cb-66815d32b8a8-kube-api-access-8c5qf\") pod \"payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd\" (UID: \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:13.627121 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.627052 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59fe046-6c9f-450a-b1cb-66815d32b8a8-plugins-config-volume\") pod \"payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd\" (UID: \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:13.637105 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.637078 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm"] Jul 28 22:33:13.640580 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.640558 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:13.650664 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.650642 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm"] Jul 28 22:33:13.728227 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.728171 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41af7f5e-056c-48f1-80b9-bbb27aa29196-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm\" (UID: \"41af7f5e-056c-48f1-80b9-bbb27aa29196\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:13.728427 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.728237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59fe046-6c9f-450a-b1cb-66815d32b8a8-plugins-config-volume\") pod \"payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd\" (UID: \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:13.728427 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.728309 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfxf\" (UniqueName: \"kubernetes.io/projected/41af7f5e-056c-48f1-80b9-bbb27aa29196-kube-api-access-srfxf\") pod \"payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm\" (UID: \"41af7f5e-056c-48f1-80b9-bbb27aa29196\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:13.728427 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.728397 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5qf\" (UniqueName: \"kubernetes.io/projected/d59fe046-6c9f-450a-b1cb-66815d32b8a8-kube-api-access-8c5qf\") pod \"payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd\" (UID: \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:13.729024 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.729004 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59fe046-6c9f-450a-b1cb-66815d32b8a8-plugins-config-volume\") pod \"payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd\" (UID: \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:13.736224 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.736196 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5qf\" (UniqueName: \"kubernetes.io/projected/d59fe046-6c9f-450a-b1cb-66815d32b8a8-kube-api-access-8c5qf\") pod \"payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd\" (UID: \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\") " pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:13.829725 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.829682 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41af7f5e-056c-48f1-80b9-bbb27aa29196-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm\" (UID: \"41af7f5e-056c-48f1-80b9-bbb27aa29196\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:13.829897 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.829778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfxf\" (UniqueName: \"kubernetes.io/projected/41af7f5e-056c-48f1-80b9-bbb27aa29196-kube-api-access-srfxf\") pod \"payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm\" (UID: \"41af7f5e-056c-48f1-80b9-bbb27aa29196\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:13.830509 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.830489 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41af7f5e-056c-48f1-80b9-bbb27aa29196-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm\" (UID: \"41af7f5e-056c-48f1-80b9-bbb27aa29196\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:13.842447 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.842412 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfxf\" (UniqueName: \"kubernetes.io/projected/41af7f5e-056c-48f1-80b9-bbb27aa29196-kube-api-access-srfxf\") pod \"payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm\" (UID: \"41af7f5e-056c-48f1-80b9-bbb27aa29196\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:13.925453 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.925351 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:13.952919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:13.952886 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:14.076006 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:14.075963 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd"] Jul 28 22:33:14.077756 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:33:14.077728 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59fe046_6c9f_450a_b1cb_66815d32b8a8.slice/crio-8f298be448e835cf829d832dbc801d07b4abd02a3713a41734e2875c70d935ad WatchSource:0}: Error finding container 8f298be448e835cf829d832dbc801d07b4abd02a3713a41734e2875c70d935ad: Status 404 returned error can't find the container with id 8f298be448e835cf829d832dbc801d07b4abd02a3713a41734e2875c70d935ad Jul 28 22:33:14.100187 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:14.100157 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm"] Jul 28 22:33:14.102146 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:33:14.102121 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41af7f5e_056c_48f1_80b9_bbb27aa29196.slice/crio-abe9209ccc0197cb08d3e8c6f3d02cc4c73b99ab83ffd49b8e7a3ad6e8363c3c WatchSource:0}: Error finding container abe9209ccc0197cb08d3e8c6f3d02cc4c73b99ab83ffd49b8e7a3ad6e8363c3c: Status 404 returned error can't find the container with id abe9209ccc0197cb08d3e8c6f3d02cc4c73b99ab83ffd49b8e7a3ad6e8363c3c Jul 28 22:33:14.229060 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:14.229023 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" event={"ID":"41af7f5e-056c-48f1-80b9-bbb27aa29196","Type":"ContainerStarted","Data":"21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade"} Jul 28 22:33:14.229286 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:14.229086 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:14.229286 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:14.229112 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" event={"ID":"41af7f5e-056c-48f1-80b9-bbb27aa29196","Type":"ContainerStarted","Data":"abe9209ccc0197cb08d3e8c6f3d02cc4c73b99ab83ffd49b8e7a3ad6e8363c3c"} Jul 28 22:33:14.230560 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:14.230536 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" event={"ID":"d59fe046-6c9f-450a-b1cb-66815d32b8a8","Type":"ContainerStarted","Data":"dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178"} Jul 28 22:33:14.230652 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:14.230583 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:14.230652 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:14.230603 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" event={"ID":"d59fe046-6c9f-450a-b1cb-66815d32b8a8","Type":"ContainerStarted","Data":"8f298be448e835cf829d832dbc801d07b4abd02a3713a41734e2875c70d935ad"} Jul 28 22:33:14.246783 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:14.246726 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" podStartSLOduration=1.246705091 podStartE2EDuration="1.246705091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:33:14.24418797 +0000 UTC m=+1718.882047099" watchObservedRunningTime="2026-07-28 22:33:14.246705091 +0000 UTC m=+1718.884564217" Jul 28 22:33:25.237692 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:25.237660 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:25.238201 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:25.237897 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:25.254801 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:25.254739 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" podStartSLOduration=12.254723528 podStartE2EDuration="12.254723528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:33:14.2643932 +0000 UTC m=+1718.902252328" watchObservedRunningTime="2026-07-28 22:33:25.254723528 +0000 UTC m=+1729.892582656" Jul 28 22:33:25.497058 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:25.496954 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:33:43.619511 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:43.619474 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd"] Jul 28 22:33:43.620055 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:43.619720 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" podUID="d59fe046-6c9f-450a-b1cb-66815d32b8a8" containerName="payload-processing" containerID="cri-o://dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178" gracePeriod=30 Jul 28 22:33:43.628077 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:43.627765 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm"] Jul 28 22:33:43.628167 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:43.628072 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" podUID="41af7f5e-056c-48f1-80b9-bbb27aa29196" containerName="payload-pre-processing" containerID="cri-o://21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade" gracePeriod=30 Jul 28 22:33:43.880480 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:43.880414 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:43.909181 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:43.909159 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:44.018757 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.018723 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59fe046-6c9f-450a-b1cb-66815d32b8a8-plugins-config-volume\") pod \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\" (UID: \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\") " Jul 28 22:33:44.018976 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.018771 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41af7f5e-056c-48f1-80b9-bbb27aa29196-plugins-config-volume\") pod \"41af7f5e-056c-48f1-80b9-bbb27aa29196\" (UID: \"41af7f5e-056c-48f1-80b9-bbb27aa29196\") " Jul 28 22:33:44.018976 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.018801 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c5qf\" (UniqueName: \"kubernetes.io/projected/d59fe046-6c9f-450a-b1cb-66815d32b8a8-kube-api-access-8c5qf\") pod \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\" (UID: \"d59fe046-6c9f-450a-b1cb-66815d32b8a8\") " Jul 28 22:33:44.018976 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.018818 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srfxf\" (UniqueName: \"kubernetes.io/projected/41af7f5e-056c-48f1-80b9-bbb27aa29196-kube-api-access-srfxf\") pod \"41af7f5e-056c-48f1-80b9-bbb27aa29196\" (UID: \"41af7f5e-056c-48f1-80b9-bbb27aa29196\") " Jul 28 22:33:44.019181 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.019157 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59fe046-6c9f-450a-b1cb-66815d32b8a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d59fe046-6c9f-450a-b1cb-66815d32b8a8" (UID: "d59fe046-6c9f-450a-b1cb-66815d32b8a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:33:44.019299 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.019224 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41af7f5e-056c-48f1-80b9-bbb27aa29196-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41af7f5e-056c-48f1-80b9-bbb27aa29196" (UID: "41af7f5e-056c-48f1-80b9-bbb27aa29196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:33:44.020964 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.020941 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59fe046-6c9f-450a-b1cb-66815d32b8a8-kube-api-access-8c5qf" (OuterVolumeSpecName: "kube-api-access-8c5qf") pod "d59fe046-6c9f-450a-b1cb-66815d32b8a8" (UID: "d59fe046-6c9f-450a-b1cb-66815d32b8a8"). InnerVolumeSpecName "kube-api-access-8c5qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:33:44.021035 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.020973 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41af7f5e-056c-48f1-80b9-bbb27aa29196-kube-api-access-srfxf" (OuterVolumeSpecName: "kube-api-access-srfxf") pod "41af7f5e-056c-48f1-80b9-bbb27aa29196" (UID: "41af7f5e-056c-48f1-80b9-bbb27aa29196"). InnerVolumeSpecName "kube-api-access-srfxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:33:44.120368 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.120330 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59fe046-6c9f-450a-b1cb-66815d32b8a8-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:33:44.120368 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.120365 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41af7f5e-056c-48f1-80b9-bbb27aa29196-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:33:44.120368 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.120374 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5qf\" (UniqueName: \"kubernetes.io/projected/d59fe046-6c9f-450a-b1cb-66815d32b8a8-kube-api-access-8c5qf\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:33:44.120596 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.120382 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srfxf\" (UniqueName: \"kubernetes.io/projected/41af7f5e-056c-48f1-80b9-bbb27aa29196-kube-api-access-srfxf\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:33:44.377868 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.377828 2557 generic.go:356] "Generic (PLEG): container finished" podID="41af7f5e-056c-48f1-80b9-bbb27aa29196" containerID="21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade" exitCode=0 Jul 28 22:33:44.378085 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.377879 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" Jul 28 22:33:44.378085 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.377945 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" event={"ID":"41af7f5e-056c-48f1-80b9-bbb27aa29196","Type":"ContainerDied","Data":"21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade"} Jul 28 22:33:44.378085 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.378002 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm" event={"ID":"41af7f5e-056c-48f1-80b9-bbb27aa29196","Type":"ContainerDied","Data":"abe9209ccc0197cb08d3e8c6f3d02cc4c73b99ab83ffd49b8e7a3ad6e8363c3c"} Jul 28 22:33:44.378085 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.378025 2557 scope.go:117] "RemoveContainer" containerID="21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade" Jul 28 22:33:44.379308 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.379284 2557 generic.go:356] "Generic (PLEG): container finished" podID="d59fe046-6c9f-450a-b1cb-66815d32b8a8" containerID="dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178" exitCode=0 Jul 28 22:33:44.379419 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.379321 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" event={"ID":"d59fe046-6c9f-450a-b1cb-66815d32b8a8","Type":"ContainerDied","Data":"dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178"} Jul 28 22:33:44.379419 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.379341 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" Jul 28 22:33:44.379419 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.379381 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd" event={"ID":"d59fe046-6c9f-450a-b1cb-66815d32b8a8","Type":"ContainerDied","Data":"8f298be448e835cf829d832dbc801d07b4abd02a3713a41734e2875c70d935ad"} Jul 28 22:33:44.387163 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.387143 2557 scope.go:117] "RemoveContainer" containerID="21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade" Jul 28 22:33:44.387416 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:33:44.387398 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade\": container with ID starting with 21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade not found: ID does not exist" containerID="21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade" Jul 28 22:33:44.387459 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.387425 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade"} err="failed to get container status \"21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade\": rpc error: code = NotFound desc = could not find container \"21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade\": container with ID starting with 21d9f31800b51f4880bb37b81226a776dbb6ffe3148975cbfc9bfe09a41b6ade not found: ID does not exist" Jul 28 22:33:44.387459 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.387441 2557 scope.go:117] "RemoveContainer" containerID="dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178" Jul 28 22:33:44.394787 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.394771 2557 scope.go:117] "RemoveContainer" containerID="dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178" Jul 28 22:33:44.395254 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:33:44.395225 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178\": container with ID starting with dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178 not found: ID does not exist" containerID="dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178" Jul 28 22:33:44.395409 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.395267 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178"} err="failed to get container status \"dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178\": rpc error: code = NotFound desc = could not find container \"dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178\": container with ID starting with dc6cf5b0d86ba15318e967a670ba1fe6ace9a7d422684f4e9ae550f2a95ff178 not found: ID does not exist" Jul 28 22:33:44.397877 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.397858 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm"] Jul 28 22:33:44.401683 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.401664 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1a8249b-7dfc86c8c6-x8lzm"] Jul 28 22:33:44.412218 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.412194 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd"] Jul 28 22:33:44.417884 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:44.417864 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e1a8249b-7ffc99f5d4-p54xd"] Jul 28 22:33:46.045497 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:46.045464 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41af7f5e-056c-48f1-80b9-bbb27aa29196" path="/var/lib/kubelet/pods/41af7f5e-056c-48f1-80b9-bbb27aa29196/volumes" Jul 28 22:33:46.045872 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:46.045832 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d59fe046-6c9f-450a-b1cb-66815d32b8a8" path="/var/lib/kubelet/pods/d59fe046-6c9f-450a-b1cb-66815d32b8a8/volumes" Jul 28 22:33:48.004365 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.004328 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:33:48.831890 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.831858 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl"] Jul 28 22:33:48.832285 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.832271 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41af7f5e-056c-48f1-80b9-bbb27aa29196" containerName="payload-pre-processing" Jul 28 22:33:48.832335 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.832287 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="41af7f5e-056c-48f1-80b9-bbb27aa29196" containerName="payload-pre-processing" Jul 28 22:33:48.832335 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.832301 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59fe046-6c9f-450a-b1cb-66815d32b8a8" containerName="payload-processing" Jul 28 22:33:48.832335 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.832306 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59fe046-6c9f-450a-b1cb-66815d32b8a8" containerName="payload-processing" Jul 28 22:33:48.832429 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.832369 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d59fe046-6c9f-450a-b1cb-66815d32b8a8" containerName="payload-processing" Jul 28 22:33:48.832429 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.832376 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="41af7f5e-056c-48f1-80b9-bbb27aa29196" containerName="payload-pre-processing" Jul 28 22:33:48.836886 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.836868 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:33:48.841436 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.841416 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f8361843-dockercfg-czl4j\"" Jul 28 22:33:48.841825 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.841809 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f8361843\"" Jul 28 22:33:48.847474 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.847447 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl"] Jul 28 22:33:48.865382 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.865352 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn"] Jul 28 22:33:48.869490 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.869466 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:33:48.874503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.874478 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn"] Jul 28 22:33:48.963831 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.963782 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7cm\" (UniqueName: \"kubernetes.io/projected/09794622-ddd6-4574-b187-feaf157f80f5-kube-api-access-gb7cm\") pod \"payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn\" (UID: \"09794622-ddd6-4574-b187-feaf157f80f5\") " pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:33:48.964263 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.964241 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631f2d03-bb80-461c-af56-31db860e7efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl\" (UID: \"631f2d03-bb80-461c-af56-31db860e7efc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:33:48.964444 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.964427 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvk45\" (UniqueName: \"kubernetes.io/projected/631f2d03-bb80-461c-af56-31db860e7efc-kube-api-access-cvk45\") pod \"payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl\" (UID: \"631f2d03-bb80-461c-af56-31db860e7efc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:33:48.964622 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:48.964608 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09794622-ddd6-4574-b187-feaf157f80f5-plugins-config-volume\") pod \"payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn\" (UID: \"09794622-ddd6-4574-b187-feaf157f80f5\") " pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:33:49.066085 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.066049 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631f2d03-bb80-461c-af56-31db860e7efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl\" (UID: \"631f2d03-bb80-461c-af56-31db860e7efc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:33:49.066616 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.066098 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvk45\" (UniqueName: \"kubernetes.io/projected/631f2d03-bb80-461c-af56-31db860e7efc-kube-api-access-cvk45\") pod \"payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl\" (UID: \"631f2d03-bb80-461c-af56-31db860e7efc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:33:49.066616 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.066178 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09794622-ddd6-4574-b187-feaf157f80f5-plugins-config-volume\") pod \"payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn\" (UID: \"09794622-ddd6-4574-b187-feaf157f80f5\") " pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:33:49.066864 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.066833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7cm\" (UniqueName: \"kubernetes.io/projected/09794622-ddd6-4574-b187-feaf157f80f5-kube-api-access-gb7cm\") pod \"payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn\" (UID: \"09794622-ddd6-4574-b187-feaf157f80f5\") " pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:33:49.067040 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.066959 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631f2d03-bb80-461c-af56-31db860e7efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl\" (UID: \"631f2d03-bb80-461c-af56-31db860e7efc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:33:49.067040 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.067022 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09794622-ddd6-4574-b187-feaf157f80f5-plugins-config-volume\") pod \"payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn\" (UID: \"09794622-ddd6-4574-b187-feaf157f80f5\") " pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:33:49.075790 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.075731 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvk45\" (UniqueName: \"kubernetes.io/projected/631f2d03-bb80-461c-af56-31db860e7efc-kube-api-access-cvk45\") pod \"payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl\" (UID: \"631f2d03-bb80-461c-af56-31db860e7efc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:33:49.076000 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.075979 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7cm\" (UniqueName: \"kubernetes.io/projected/09794622-ddd6-4574-b187-feaf157f80f5-kube-api-access-gb7cm\") pod \"payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn\" (UID: \"09794622-ddd6-4574-b187-feaf157f80f5\") " pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:33:49.148739 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.148644 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:33:49.183906 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.183869 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:33:49.298227 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.298197 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl"] Jul 28 22:33:49.301456 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:33:49.301427 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631f2d03_bb80_461c_af56_31db860e7efc.slice/crio-116e0f5d15030b7522e5b685b12aaf265f264b96c353ea6840d403dfbb3bf24b WatchSource:0}: Error finding container 116e0f5d15030b7522e5b685b12aaf265f264b96c353ea6840d403dfbb3bf24b: Status 404 returned error can't find the container with id 116e0f5d15030b7522e5b685b12aaf265f264b96c353ea6840d403dfbb3bf24b Jul 28 22:33:49.333607 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.333584 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn"] Jul 28 22:33:49.335753 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:33:49.335727 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09794622_ddd6_4574_b187_feaf157f80f5.slice/crio-08ff780b709e18a8d9fae0bf9b3b4df31902c3723756b3cee18370df52276591 WatchSource:0}: Error finding container 08ff780b709e18a8d9fae0bf9b3b4df31902c3723756b3cee18370df52276591: Status 404 returned error can't find the container with id 08ff780b709e18a8d9fae0bf9b3b4df31902c3723756b3cee18370df52276591 Jul 28 22:33:49.410139 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.410057 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" event={"ID":"09794622-ddd6-4574-b187-feaf157f80f5","Type":"ContainerStarted","Data":"90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8"} Jul 28 22:33:49.410139 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.410114 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" event={"ID":"09794622-ddd6-4574-b187-feaf157f80f5","Type":"ContainerStarted","Data":"08ff780b709e18a8d9fae0bf9b3b4df31902c3723756b3cee18370df52276591"} Jul 28 22:33:49.410139 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.410138 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:33:49.411711 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.411687 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" event={"ID":"631f2d03-bb80-461c-af56-31db860e7efc","Type":"ContainerStarted","Data":"4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31"} Jul 28 22:33:49.411832 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.411734 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:33:49.411832 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.411750 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" event={"ID":"631f2d03-bb80-461c-af56-31db860e7efc","Type":"ContainerStarted","Data":"116e0f5d15030b7522e5b685b12aaf265f264b96c353ea6840d403dfbb3bf24b"} Jul 28 22:33:49.427298 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.427245 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" podStartSLOduration=1.42722938 podStartE2EDuration="1.42722938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:33:49.426220058 +0000 UTC m=+1754.064079187" watchObservedRunningTime="2026-07-28 22:33:49.42722938 +0000 UTC m=+1754.065088487" Jul 28 22:33:49.444491 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:49.444442 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" podStartSLOduration=1.444423116 podStartE2EDuration="1.444423116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:33:49.442265465 +0000 UTC m=+1754.080124593" watchObservedRunningTime="2026-07-28 22:33:49.444423116 +0000 UTC m=+1754.082282245" Jul 28 22:33:59.690728 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:33:59.690689 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:34:00.418527 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:00.418486 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:34:00.418717 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:00.418569 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:34:18.681017 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:18.680985 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn"] Jul 28 22:34:18.681522 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:18.681194 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" podUID="09794622-ddd6-4574-b187-feaf157f80f5" containerName="payload-processing" containerID="cri-o://90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8" gracePeriod=30 Jul 28 22:34:18.688731 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:18.688701 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl"] Jul 28 22:34:18.689009 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:18.688956 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" podUID="631f2d03-bb80-461c-af56-31db860e7efc" containerName="payload-pre-processing" containerID="cri-o://4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31" gracePeriod=30 Jul 28 22:34:18.970689 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:18.970664 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:34:18.973857 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:18.973837 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:34:19.044724 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.044689 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09794622-ddd6-4574-b187-feaf157f80f5-plugins-config-volume\") pod \"09794622-ddd6-4574-b187-feaf157f80f5\" (UID: \"09794622-ddd6-4574-b187-feaf157f80f5\") " Jul 28 22:34:19.045000 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.044792 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvk45\" (UniqueName: \"kubernetes.io/projected/631f2d03-bb80-461c-af56-31db860e7efc-kube-api-access-cvk45\") pod \"631f2d03-bb80-461c-af56-31db860e7efc\" (UID: \"631f2d03-bb80-461c-af56-31db860e7efc\") " Jul 28 22:34:19.045000 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.044821 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631f2d03-bb80-461c-af56-31db860e7efc-plugins-config-volume\") pod \"631f2d03-bb80-461c-af56-31db860e7efc\" (UID: \"631f2d03-bb80-461c-af56-31db860e7efc\") " Jul 28 22:34:19.045000 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.044885 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb7cm\" (UniqueName: \"kubernetes.io/projected/09794622-ddd6-4574-b187-feaf157f80f5-kube-api-access-gb7cm\") pod \"09794622-ddd6-4574-b187-feaf157f80f5\" (UID: \"09794622-ddd6-4574-b187-feaf157f80f5\") " Jul 28 22:34:19.045205 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.045168 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09794622-ddd6-4574-b187-feaf157f80f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09794622-ddd6-4574-b187-feaf157f80f5" (UID: "09794622-ddd6-4574-b187-feaf157f80f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:34:19.045271 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.045250 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631f2d03-bb80-461c-af56-31db860e7efc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "631f2d03-bb80-461c-af56-31db860e7efc" (UID: "631f2d03-bb80-461c-af56-31db860e7efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:34:19.047363 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.047319 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09794622-ddd6-4574-b187-feaf157f80f5-kube-api-access-gb7cm" (OuterVolumeSpecName: "kube-api-access-gb7cm") pod "09794622-ddd6-4574-b187-feaf157f80f5" (UID: "09794622-ddd6-4574-b187-feaf157f80f5"). InnerVolumeSpecName "kube-api-access-gb7cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:34:19.047578 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.047557 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631f2d03-bb80-461c-af56-31db860e7efc-kube-api-access-cvk45" (OuterVolumeSpecName: "kube-api-access-cvk45") pod "631f2d03-bb80-461c-af56-31db860e7efc" (UID: "631f2d03-bb80-461c-af56-31db860e7efc"). InnerVolumeSpecName "kube-api-access-cvk45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:34:19.145815 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.145783 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvk45\" (UniqueName: \"kubernetes.io/projected/631f2d03-bb80-461c-af56-31db860e7efc-kube-api-access-cvk45\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:34:19.145815 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.145812 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631f2d03-bb80-461c-af56-31db860e7efc-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:34:19.146050 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.145828 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7cm\" (UniqueName: \"kubernetes.io/projected/09794622-ddd6-4574-b187-feaf157f80f5-kube-api-access-gb7cm\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:34:19.146050 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.145840 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09794622-ddd6-4574-b187-feaf157f80f5-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:34:19.560369 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.560333 2557 generic.go:356] "Generic (PLEG): container finished" podID="09794622-ddd6-4574-b187-feaf157f80f5" containerID="90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8" exitCode=0 Jul 28 22:34:19.560561 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.560377 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" Jul 28 22:34:19.560561 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.560425 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" event={"ID":"09794622-ddd6-4574-b187-feaf157f80f5","Type":"ContainerDied","Data":"90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8"} Jul 28 22:34:19.560561 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.560482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn" event={"ID":"09794622-ddd6-4574-b187-feaf157f80f5","Type":"ContainerDied","Data":"08ff780b709e18a8d9fae0bf9b3b4df31902c3723756b3cee18370df52276591"} Jul 28 22:34:19.560561 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.560526 2557 scope.go:117] "RemoveContainer" containerID="90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8" Jul 28 22:34:19.561783 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.561758 2557 generic.go:356] "Generic (PLEG): container finished" podID="631f2d03-bb80-461c-af56-31db860e7efc" containerID="4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31" exitCode=0 Jul 28 22:34:19.561882 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.561802 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" event={"ID":"631f2d03-bb80-461c-af56-31db860e7efc","Type":"ContainerDied","Data":"4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31"} Jul 28 22:34:19.561882 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.561818 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" Jul 28 22:34:19.561882 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.561845 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl" event={"ID":"631f2d03-bb80-461c-af56-31db860e7efc","Type":"ContainerDied","Data":"116e0f5d15030b7522e5b685b12aaf265f264b96c353ea6840d403dfbb3bf24b"} Jul 28 22:34:19.571133 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.571055 2557 scope.go:117] "RemoveContainer" containerID="90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8" Jul 28 22:34:19.571430 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:34:19.571391 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8\": container with ID starting with 90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8 not found: ID does not exist" containerID="90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8" Jul 28 22:34:19.571430 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.571417 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8"} err="failed to get container status \"90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8\": rpc error: code = NotFound desc = could not find container \"90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8\": container with ID starting with 90c0f343c466c2f97b147135048b48adbb24a873bdfdb38b27d9a352cd447fb8 not found: ID does not exist" Jul 28 22:34:19.571623 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.571435 2557 scope.go:117] "RemoveContainer" containerID="4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31" Jul 28 22:34:19.579712 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.579697 2557 scope.go:117] "RemoveContainer" containerID="4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31" Jul 28 22:34:19.580037 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:34:19.580018 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31\": container with ID starting with 4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31 not found: ID does not exist" containerID="4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31" Jul 28 22:34:19.580126 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.580047 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31"} err="failed to get container status \"4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31\": rpc error: code = NotFound desc = could not find container \"4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31\": container with ID starting with 4dec2704213e59737d501c2abb5b1f16dcc477b7f66d0b9f8f15680ffa07ec31 not found: ID does not exist" Jul 28 22:34:19.585477 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.585455 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn"] Jul 28 22:34:19.590875 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.590852 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f8361843-b94d84fbb-zx6tn"] Jul 28 22:34:19.601081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.601051 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl"] Jul 28 22:34:19.611869 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:19.611835 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f8361843-6455776868-6cnpl"] Jul 28 22:34:20.045705 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:20.045673 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09794622-ddd6-4574-b187-feaf157f80f5" path="/var/lib/kubelet/pods/09794622-ddd6-4574-b187-feaf157f80f5/volumes" Jul 28 22:34:20.046113 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:20.046062 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="631f2d03-bb80-461c-af56-31db860e7efc" path="/var/lib/kubelet/pods/631f2d03-bb80-461c-af56-31db860e7efc/volumes" Jul 28 22:34:22.794806 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:22.794775 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:34:32.386203 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:34:32.386121 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:35:44.400273 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:35:44.400233 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:35:59.992715 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:35:59.992630 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:36:09.698570 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:09.698531 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:36:16.162985 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.162944 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq"] Jul 28 22:36:16.163563 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.163537 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09794622-ddd6-4574-b187-feaf157f80f5" containerName="payload-processing" Jul 28 22:36:16.163563 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.163558 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="09794622-ddd6-4574-b187-feaf157f80f5" containerName="payload-processing" Jul 28 22:36:16.163682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.163585 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631f2d03-bb80-461c-af56-31db860e7efc" containerName="payload-pre-processing" Jul 28 22:36:16.163682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.163594 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="631f2d03-bb80-461c-af56-31db860e7efc" containerName="payload-pre-processing" Jul 28 22:36:16.163682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.163676 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="631f2d03-bb80-461c-af56-31db860e7efc" containerName="payload-pre-processing" Jul 28 22:36:16.163820 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.163695 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="09794622-ddd6-4574-b187-feaf157f80f5" containerName="payload-processing" Jul 28 22:36:16.166797 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.166774 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:16.169415 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.169392 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e64758c0\"" Jul 28 22:36:16.177535 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.177505 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq"] Jul 28 22:36:16.185088 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.185062 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw"] Jul 28 22:36:16.188688 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.188655 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:16.191488 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.191465 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e64758c0-dockercfg-vwscd\"" Jul 28 22:36:16.198671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.198644 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw"] Jul 28 22:36:16.232096 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.232057 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tn86\" (UniqueName: \"kubernetes.io/projected/01712273-4105-4895-b682-b8cd22531e5f-kube-api-access-9tn86\") pod \"payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq\" (UID: \"01712273-4105-4895-b682-b8cd22531e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:16.232297 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.232113 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01712273-4105-4895-b682-b8cd22531e5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq\" (UID: \"01712273-4105-4895-b682-b8cd22531e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:16.232297 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.232137 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0257ae65-5c96-45ec-ac02-cd108b2151e5-plugins-config-volume\") pod \"payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw\" (UID: \"0257ae65-5c96-45ec-ac02-cd108b2151e5\") " pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:16.232394 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.232298 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nr56\" (UniqueName: \"kubernetes.io/projected/0257ae65-5c96-45ec-ac02-cd108b2151e5-kube-api-access-9nr56\") pod \"payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw\" (UID: \"0257ae65-5c96-45ec-ac02-cd108b2151e5\") " pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:16.333265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.333231 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nr56\" (UniqueName: \"kubernetes.io/projected/0257ae65-5c96-45ec-ac02-cd108b2151e5-kube-api-access-9nr56\") pod \"payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw\" (UID: \"0257ae65-5c96-45ec-ac02-cd108b2151e5\") " pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:16.333468 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.333280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tn86\" (UniqueName: \"kubernetes.io/projected/01712273-4105-4895-b682-b8cd22531e5f-kube-api-access-9tn86\") pod \"payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq\" (UID: \"01712273-4105-4895-b682-b8cd22531e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:16.333468 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.333316 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01712273-4105-4895-b682-b8cd22531e5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq\" (UID: \"01712273-4105-4895-b682-b8cd22531e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:16.333468 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.333337 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0257ae65-5c96-45ec-ac02-cd108b2151e5-plugins-config-volume\") pod \"payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw\" (UID: \"0257ae65-5c96-45ec-ac02-cd108b2151e5\") " pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:16.334076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.334049 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01712273-4105-4895-b682-b8cd22531e5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq\" (UID: \"01712273-4105-4895-b682-b8cd22531e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:16.334186 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.334138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0257ae65-5c96-45ec-ac02-cd108b2151e5-plugins-config-volume\") pod \"payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw\" (UID: \"0257ae65-5c96-45ec-ac02-cd108b2151e5\") " pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:16.341416 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.341377 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tn86\" (UniqueName: \"kubernetes.io/projected/01712273-4105-4895-b682-b8cd22531e5f-kube-api-access-9tn86\") pod \"payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq\" (UID: \"01712273-4105-4895-b682-b8cd22531e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:16.341531 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.341438 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nr56\" (UniqueName: \"kubernetes.io/projected/0257ae65-5c96-45ec-ac02-cd108b2151e5-kube-api-access-9nr56\") pod \"payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw\" (UID: \"0257ae65-5c96-45ec-ac02-cd108b2151e5\") " pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:16.480070 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.480037 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:16.500682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.500649 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:16.632436 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.632409 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq"] Jul 28 22:36:16.634605 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:36:16.634576 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01712273_4105_4895_b682_b8cd22531e5f.slice/crio-30d4d7e881d97e33fd94c2510886ead1d1dd121593ea8f820b57c7b3e6166927 WatchSource:0}: Error finding container 30d4d7e881d97e33fd94c2510886ead1d1dd121593ea8f820b57c7b3e6166927: Status 404 returned error can't find the container with id 30d4d7e881d97e33fd94c2510886ead1d1dd121593ea8f820b57c7b3e6166927 Jul 28 22:36:16.650831 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:16.650791 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw"] Jul 28 22:36:16.652796 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:36:16.652727 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0257ae65_5c96_45ec_ac02_cd108b2151e5.slice/crio-bc359cd73b1f131dc863b708bbc34c50c8ba6e0835d61d7f717cc3a14d3ddd94 WatchSource:0}: Error finding container bc359cd73b1f131dc863b708bbc34c50c8ba6e0835d61d7f717cc3a14d3ddd94: Status 404 returned error can't find the container with id bc359cd73b1f131dc863b708bbc34c50c8ba6e0835d61d7f717cc3a14d3ddd94 Jul 28 22:36:17.130751 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:17.130712 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" event={"ID":"0257ae65-5c96-45ec-ac02-cd108b2151e5","Type":"ContainerStarted","Data":"14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6"} Jul 28 22:36:17.130973 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:17.130778 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" event={"ID":"0257ae65-5c96-45ec-ac02-cd108b2151e5","Type":"ContainerStarted","Data":"bc359cd73b1f131dc863b708bbc34c50c8ba6e0835d61d7f717cc3a14d3ddd94"} Jul 28 22:36:17.130973 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:17.130807 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:17.132347 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:17.132310 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" event={"ID":"01712273-4105-4895-b682-b8cd22531e5f","Type":"ContainerStarted","Data":"9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1"} Jul 28 22:36:17.132477 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:17.132361 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" event={"ID":"01712273-4105-4895-b682-b8cd22531e5f","Type":"ContainerStarted","Data":"30d4d7e881d97e33fd94c2510886ead1d1dd121593ea8f820b57c7b3e6166927"} Jul 28 22:36:17.132477 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:17.132420 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:17.150704 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:17.150653 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" podStartSLOduration=1.15063559 podStartE2EDuration="1.15063559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:17.148004322 +0000 UTC m=+1901.785863456" watchObservedRunningTime="2026-07-28 22:36:17.15063559 +0000 UTC m=+1901.788494718" Jul 28 22:36:17.164595 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:17.164540 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" podStartSLOduration=1.164520274 podStartE2EDuration="1.164520274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:17.163614657 +0000 UTC m=+1901.801473786" watchObservedRunningTime="2026-07-28 22:36:17.164520274 +0000 UTC m=+1901.802379403" Jul 28 22:36:19.196583 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:19.196545 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:36:28.139727 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:28.139695 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:28.140144 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:28.139764 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:58.528457 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.528423 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw"] Jul 28 22:36:58.529073 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.528677 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" podUID="0257ae65-5c96-45ec-ac02-cd108b2151e5" containerName="payload-processing" containerID="cri-o://14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6" gracePeriod=30 Jul 28 22:36:58.537824 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.537784 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq"] Jul 28 22:36:58.538037 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.538009 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" podUID="01712273-4105-4895-b682-b8cd22531e5f" containerName="payload-pre-processing" containerID="cri-o://9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1" gracePeriod=30 Jul 28 22:36:58.819584 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.819560 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:58.822297 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.822275 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:58.938391 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.938353 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nr56\" (UniqueName: \"kubernetes.io/projected/0257ae65-5c96-45ec-ac02-cd108b2151e5-kube-api-access-9nr56\") pod \"0257ae65-5c96-45ec-ac02-cd108b2151e5\" (UID: \"0257ae65-5c96-45ec-ac02-cd108b2151e5\") " Jul 28 22:36:58.938563 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.938409 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0257ae65-5c96-45ec-ac02-cd108b2151e5-plugins-config-volume\") pod \"0257ae65-5c96-45ec-ac02-cd108b2151e5\" (UID: \"0257ae65-5c96-45ec-ac02-cd108b2151e5\") " Jul 28 22:36:58.938563 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.938445 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01712273-4105-4895-b682-b8cd22531e5f-plugins-config-volume\") pod \"01712273-4105-4895-b682-b8cd22531e5f\" (UID: \"01712273-4105-4895-b682-b8cd22531e5f\") " Jul 28 22:36:58.938563 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.938528 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tn86\" (UniqueName: \"kubernetes.io/projected/01712273-4105-4895-b682-b8cd22531e5f-kube-api-access-9tn86\") pod \"01712273-4105-4895-b682-b8cd22531e5f\" (UID: \"01712273-4105-4895-b682-b8cd22531e5f\") " Jul 28 22:36:58.938832 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.938805 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01712273-4105-4895-b682-b8cd22531e5f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01712273-4105-4895-b682-b8cd22531e5f" (UID: "01712273-4105-4895-b682-b8cd22531e5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:36:58.938914 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.938832 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0257ae65-5c96-45ec-ac02-cd108b2151e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0257ae65-5c96-45ec-ac02-cd108b2151e5" (UID: "0257ae65-5c96-45ec-ac02-cd108b2151e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:36:58.940456 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.940434 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0257ae65-5c96-45ec-ac02-cd108b2151e5-kube-api-access-9nr56" (OuterVolumeSpecName: "kube-api-access-9nr56") pod "0257ae65-5c96-45ec-ac02-cd108b2151e5" (UID: "0257ae65-5c96-45ec-ac02-cd108b2151e5"). InnerVolumeSpecName "kube-api-access-9nr56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:36:58.940582 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:58.940561 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01712273-4105-4895-b682-b8cd22531e5f-kube-api-access-9tn86" (OuterVolumeSpecName: "kube-api-access-9tn86") pod "01712273-4105-4895-b682-b8cd22531e5f" (UID: "01712273-4105-4895-b682-b8cd22531e5f"). InnerVolumeSpecName "kube-api-access-9tn86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:36:59.040103 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.040065 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nr56\" (UniqueName: \"kubernetes.io/projected/0257ae65-5c96-45ec-ac02-cd108b2151e5-kube-api-access-9nr56\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:36:59.040103 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.040095 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0257ae65-5c96-45ec-ac02-cd108b2151e5-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:36:59.040103 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.040108 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01712273-4105-4895-b682-b8cd22531e5f-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:36:59.040341 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.040117 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tn86\" (UniqueName: \"kubernetes.io/projected/01712273-4105-4895-b682-b8cd22531e5f-kube-api-access-9tn86\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:36:59.342358 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.342325 2557 generic.go:356] "Generic (PLEG): container finished" podID="0257ae65-5c96-45ec-ac02-cd108b2151e5" containerID="14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6" exitCode=0 Jul 28 22:36:59.342572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.342372 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" Jul 28 22:36:59.342572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.342417 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" event={"ID":"0257ae65-5c96-45ec-ac02-cd108b2151e5","Type":"ContainerDied","Data":"14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6"} Jul 28 22:36:59.342572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.342480 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw" event={"ID":"0257ae65-5c96-45ec-ac02-cd108b2151e5","Type":"ContainerDied","Data":"bc359cd73b1f131dc863b708bbc34c50c8ba6e0835d61d7f717cc3a14d3ddd94"} Jul 28 22:36:59.342572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.342510 2557 scope.go:117] "RemoveContainer" containerID="14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6" Jul 28 22:36:59.343775 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.343749 2557 generic.go:356] "Generic (PLEG): container finished" podID="01712273-4105-4895-b682-b8cd22531e5f" containerID="9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1" exitCode=0 Jul 28 22:36:59.343855 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.343781 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" Jul 28 22:36:59.343855 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.343800 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" event={"ID":"01712273-4105-4895-b682-b8cd22531e5f","Type":"ContainerDied","Data":"9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1"} Jul 28 22:36:59.343855 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.343836 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq" event={"ID":"01712273-4105-4895-b682-b8cd22531e5f","Type":"ContainerDied","Data":"30d4d7e881d97e33fd94c2510886ead1d1dd121593ea8f820b57c7b3e6166927"} Jul 28 22:36:59.352255 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.352231 2557 scope.go:117] "RemoveContainer" containerID="14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6" Jul 28 22:36:59.352507 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:36:59.352489 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6\": container with ID starting with 14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6 not found: ID does not exist" containerID="14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6" Jul 28 22:36:59.352568 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.352515 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6"} err="failed to get container status \"14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6\": rpc error: code = NotFound desc = could not find container \"14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6\": container with ID starting with 14f156b7ac5e40b08e8c084f5aa2d94a5de4954622945f826d75af530bba77f6 not found: ID does not exist" Jul 28 22:36:59.352568 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.352531 2557 scope.go:117] "RemoveContainer" containerID="9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1" Jul 28 22:36:59.360812 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.360793 2557 scope.go:117] "RemoveContainer" containerID="9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1" Jul 28 22:36:59.361085 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:36:59.361068 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1\": container with ID starting with 9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1 not found: ID does not exist" containerID="9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1" Jul 28 22:36:59.361142 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.361095 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1"} err="failed to get container status \"9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1\": rpc error: code = NotFound desc = could not find container \"9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1\": container with ID starting with 9ce3cd8696923da06c03272fd2c4ebc4e25cb4ad42088176da92aeb9fd3b63d1 not found: ID does not exist" Jul 28 22:36:59.366544 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.366520 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq"] Jul 28 22:36:59.370847 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.370819 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e64758c0-fc4f7b448-h8nbq"] Jul 28 22:36:59.380709 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.380675 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw"] Jul 28 22:36:59.385402 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:36:59.385377 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e64758c0-5bdc9fb88b-4vrbw"] Jul 28 22:37:00.045716 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:00.045683 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01712273-4105-4895-b682-b8cd22531e5f" path="/var/lib/kubelet/pods/01712273-4105-4895-b682-b8cd22531e5f/volumes" Jul 28 22:37:00.046125 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:00.046068 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0257ae65-5c96-45ec-ac02-cd108b2151e5" path="/var/lib/kubelet/pods/0257ae65-5c96-45ec-ac02-cd108b2151e5/volumes" Jul 28 22:37:01.887676 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:01.887640 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:37:29.472448 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.472368 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx"] Jul 28 22:37:29.472858 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.472843 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0257ae65-5c96-45ec-ac02-cd108b2151e5" containerName="payload-processing" Jul 28 22:37:29.472918 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.472860 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0257ae65-5c96-45ec-ac02-cd108b2151e5" containerName="payload-processing" Jul 28 22:37:29.472918 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.472871 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01712273-4105-4895-b682-b8cd22531e5f" containerName="payload-pre-processing" Jul 28 22:37:29.472918 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.472880 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="01712273-4105-4895-b682-b8cd22531e5f" containerName="payload-pre-processing" Jul 28 22:37:29.473108 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.473008 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="01712273-4105-4895-b682-b8cd22531e5f" containerName="payload-pre-processing" Jul 28 22:37:29.473108 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.473021 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0257ae65-5c96-45ec-ac02-cd108b2151e5" containerName="payload-processing" Jul 28 22:37:29.476430 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.476408 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24"] Jul 28 22:37:29.476539 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.476521 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:29.479703 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.479686 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-01e687c7\"" Jul 28 22:37:29.479808 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.479705 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-01e687c7-dockercfg-2gssn\"" Jul 28 22:37:29.479892 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.479877 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:29.489569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.489547 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24"] Jul 28 22:37:29.492099 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.492076 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx"] Jul 28 22:37:29.510471 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.510445 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-plugins-config-volume\") pod \"payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24\" (UID: \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\") " pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:29.510584 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.510478 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jrm\" (UniqueName: \"kubernetes.io/projected/e7190a5a-56ba-4d6b-865b-8fe988b39f06-kube-api-access-g7jrm\") pod \"payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx\" (UID: \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:29.510584 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.510504 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8s4\" (UniqueName: \"kubernetes.io/projected/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-kube-api-access-kz8s4\") pod \"payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24\" (UID: \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\") " pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:29.510696 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.510603 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7190a5a-56ba-4d6b-865b-8fe988b39f06-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx\" (UID: \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:29.611278 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.611242 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-plugins-config-volume\") pod \"payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24\" (UID: \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\") " pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:29.611278 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.611283 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jrm\" (UniqueName: \"kubernetes.io/projected/e7190a5a-56ba-4d6b-865b-8fe988b39f06-kube-api-access-g7jrm\") pod \"payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx\" (UID: \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:29.611530 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.611307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8s4\" (UniqueName: \"kubernetes.io/projected/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-kube-api-access-kz8s4\") pod \"payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24\" (UID: \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\") " pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:29.611530 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.611346 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7190a5a-56ba-4d6b-865b-8fe988b39f06-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx\" (UID: \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:29.612056 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.612037 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7190a5a-56ba-4d6b-865b-8fe988b39f06-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx\" (UID: \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:29.612056 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.612048 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-plugins-config-volume\") pod \"payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24\" (UID: \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\") " pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:29.619679 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.619645 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8s4\" (UniqueName: \"kubernetes.io/projected/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-kube-api-access-kz8s4\") pod \"payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24\" (UID: \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\") " pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:29.619971 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.619952 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jrm\" (UniqueName: \"kubernetes.io/projected/e7190a5a-56ba-4d6b-865b-8fe988b39f06-kube-api-access-g7jrm\") pod \"payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx\" (UID: \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:29.789557 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.789480 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:29.797612 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.797594 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:29.942465 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.942424 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx"] Jul 28 22:37:29.944187 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:37:29.944162 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7190a5a_56ba_4d6b_865b_8fe988b39f06.slice/crio-51e30482baf73943ad776b6f3369646714ce410e979fa879069580c86e2e43d8 WatchSource:0}: Error finding container 51e30482baf73943ad776b6f3369646714ce410e979fa879069580c86e2e43d8: Status 404 returned error can't find the container with id 51e30482baf73943ad776b6f3369646714ce410e979fa879069580c86e2e43d8 Jul 28 22:37:29.963154 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:29.963072 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24"] Jul 28 22:37:29.966877 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:37:29.966849 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80372bf4_853c_4b45_8cb9_b5b94d3ac0c4.slice/crio-782159797e0cadd90e908509aea3986e2a64d345a6859cf8dde6e8a91607ca61 WatchSource:0}: Error finding container 782159797e0cadd90e908509aea3986e2a64d345a6859cf8dde6e8a91607ca61: Status 404 returned error can't find the container with id 782159797e0cadd90e908509aea3986e2a64d345a6859cf8dde6e8a91607ca61 Jul 28 22:37:30.503097 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:30.503057 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" event={"ID":"e7190a5a-56ba-4d6b-865b-8fe988b39f06","Type":"ContainerStarted","Data":"1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16"} Jul 28 22:37:30.503559 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:30.503122 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" event={"ID":"e7190a5a-56ba-4d6b-865b-8fe988b39f06","Type":"ContainerStarted","Data":"51e30482baf73943ad776b6f3369646714ce410e979fa879069580c86e2e43d8"} Jul 28 22:37:30.503559 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:30.503150 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:30.504665 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:30.504638 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" event={"ID":"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4","Type":"ContainerStarted","Data":"0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25"} Jul 28 22:37:30.504777 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:30.504689 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" event={"ID":"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4","Type":"ContainerStarted","Data":"782159797e0cadd90e908509aea3986e2a64d345a6859cf8dde6e8a91607ca61"} Jul 28 22:37:30.504777 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:30.504715 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:30.536038 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:30.535990 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" podStartSLOduration=1.5359737039999999 podStartE2EDuration="1.535973704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:37:30.534749365 +0000 UTC m=+1975.172608507" watchObservedRunningTime="2026-07-28 22:37:30.535973704 +0000 UTC m=+1975.173832832" Jul 28 22:37:30.560884 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:30.560828 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" podStartSLOduration=1.560812178 podStartE2EDuration="1.560812178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:37:30.559246172 +0000 UTC m=+1975.197105301" watchObservedRunningTime="2026-07-28 22:37:30.560812178 +0000 UTC m=+1975.198671307" Jul 28 22:37:33.201569 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:33.201534 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:37:41.512256 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:41.512225 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:37:41.512643 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:41.512299 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:37:59.596226 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:37:59.596189 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:38:18.604219 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:18.604181 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:38:36.898227 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:36.898190 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:38:38.362550 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.362512 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24"] Jul 28 22:38:38.363057 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.362774 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" podUID="80372bf4-853c-4b45-8cb9-b5b94d3ac0c4" containerName="payload-processing" containerID="cri-o://0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25" gracePeriod=30 Jul 28 22:38:38.368887 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.368854 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx"] Jul 28 22:38:38.369131 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.369100 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" podUID="e7190a5a-56ba-4d6b-865b-8fe988b39f06" containerName="payload-pre-processing" containerID="cri-o://1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16" gracePeriod=30 Jul 28 22:38:38.628159 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.628083 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:38:38.659642 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.659608 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:38:38.767243 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.767203 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7190a5a-56ba-4d6b-865b-8fe988b39f06-plugins-config-volume\") pod \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\" (UID: \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\") " Jul 28 22:38:38.767430 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.767273 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7jrm\" (UniqueName: \"kubernetes.io/projected/e7190a5a-56ba-4d6b-865b-8fe988b39f06-kube-api-access-g7jrm\") pod \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\" (UID: \"e7190a5a-56ba-4d6b-865b-8fe988b39f06\") " Jul 28 22:38:38.767430 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.767312 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-plugins-config-volume\") pod \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\" (UID: \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\") " Jul 28 22:38:38.767430 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.767334 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz8s4\" (UniqueName: \"kubernetes.io/projected/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-kube-api-access-kz8s4\") pod \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\" (UID: \"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4\") " Jul 28 22:38:38.767686 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.767655 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80372bf4-853c-4b45-8cb9-b5b94d3ac0c4" (UID: "80372bf4-853c-4b45-8cb9-b5b94d3ac0c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:38:38.767745 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.767669 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7190a5a-56ba-4d6b-865b-8fe988b39f06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7190a5a-56ba-4d6b-865b-8fe988b39f06" (UID: "e7190a5a-56ba-4d6b-865b-8fe988b39f06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:38:38.769454 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.769417 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-kube-api-access-kz8s4" (OuterVolumeSpecName: "kube-api-access-kz8s4") pod "80372bf4-853c-4b45-8cb9-b5b94d3ac0c4" (UID: "80372bf4-853c-4b45-8cb9-b5b94d3ac0c4"). InnerVolumeSpecName "kube-api-access-kz8s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:38:38.769454 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.769429 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7190a5a-56ba-4d6b-865b-8fe988b39f06-kube-api-access-g7jrm" (OuterVolumeSpecName: "kube-api-access-g7jrm") pod "e7190a5a-56ba-4d6b-865b-8fe988b39f06" (UID: "e7190a5a-56ba-4d6b-865b-8fe988b39f06"). InnerVolumeSpecName "kube-api-access-g7jrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:38:38.842990 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.842919 2557 generic.go:356] "Generic (PLEG): container finished" podID="e7190a5a-56ba-4d6b-865b-8fe988b39f06" containerID="1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16" exitCode=0 Jul 28 22:38:38.843173 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.843001 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" Jul 28 22:38:38.843173 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.843018 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" event={"ID":"e7190a5a-56ba-4d6b-865b-8fe988b39f06","Type":"ContainerDied","Data":"1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16"} Jul 28 22:38:38.843173 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.843095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx" event={"ID":"e7190a5a-56ba-4d6b-865b-8fe988b39f06","Type":"ContainerDied","Data":"51e30482baf73943ad776b6f3369646714ce410e979fa879069580c86e2e43d8"} Jul 28 22:38:38.843173 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.843124 2557 scope.go:117] "RemoveContainer" containerID="1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16" Jul 28 22:38:38.844462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.844438 2557 generic.go:356] "Generic (PLEG): container finished" podID="80372bf4-853c-4b45-8cb9-b5b94d3ac0c4" containerID="0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25" exitCode=0 Jul 28 22:38:38.844551 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.844488 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" Jul 28 22:38:38.844551 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.844491 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" event={"ID":"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4","Type":"ContainerDied","Data":"0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25"} Jul 28 22:38:38.844551 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.844532 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24" event={"ID":"80372bf4-853c-4b45-8cb9-b5b94d3ac0c4","Type":"ContainerDied","Data":"782159797e0cadd90e908509aea3986e2a64d345a6859cf8dde6e8a91607ca61"} Jul 28 22:38:38.853001 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.852976 2557 scope.go:117] "RemoveContainer" containerID="1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16" Jul 28 22:38:38.853631 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:38:38.853611 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16\": container with ID starting with 1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16 not found: ID does not exist" containerID="1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16" Jul 28 22:38:38.853840 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.853638 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16"} err="failed to get container status \"1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16\": rpc error: code = NotFound desc = could not find container \"1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16\": container with ID starting with 1b042d4dec735015ace5bdc5d6d6b8135f668ae213dec545df2d3475e2105c16 not found: ID does not exist" Jul 28 22:38:38.853840 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.853661 2557 scope.go:117] "RemoveContainer" containerID="0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25" Jul 28 22:38:38.861432 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.861414 2557 scope.go:117] "RemoveContainer" containerID="0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25" Jul 28 22:38:38.861688 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:38:38.861670 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25\": container with ID starting with 0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25 not found: ID does not exist" containerID="0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25" Jul 28 22:38:38.861754 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.861701 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25"} err="failed to get container status \"0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25\": rpc error: code = NotFound desc = could not find container \"0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25\": container with ID starting with 0af4a7e8f2044d4b2ea44a2700d41913e91a9714b714a0cb79095ecd39edaa25 not found: ID does not exist" Jul 28 22:38:38.867771 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.867746 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx"] Jul 28 22:38:38.868185 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.868166 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7jrm\" (UniqueName: \"kubernetes.io/projected/e7190a5a-56ba-4d6b-865b-8fe988b39f06-kube-api-access-g7jrm\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:38:38.868238 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.868188 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:38:38.868238 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.868199 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz8s4\" (UniqueName: \"kubernetes.io/projected/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4-kube-api-access-kz8s4\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:38:38.868238 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.868208 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7190a5a-56ba-4d6b-865b-8fe988b39f06-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:38:38.871384 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.871363 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01e687c7-868996b46d-pd7qx"] Jul 28 22:38:38.882027 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.881969 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24"] Jul 28 22:38:38.885396 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:38.885374 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01e687c7-6ddbd746c4-8vd24"] Jul 28 22:38:40.046652 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:40.046621 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80372bf4-853c-4b45-8cb9-b5b94d3ac0c4" path="/var/lib/kubelet/pods/80372bf4-853c-4b45-8cb9-b5b94d3ac0c4/volumes" Jul 28 22:38:40.047054 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:40.046989 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7190a5a-56ba-4d6b-865b-8fe988b39f06" path="/var/lib/kubelet/pods/e7190a5a-56ba-4d6b-865b-8fe988b39f06/volumes" Jul 28 22:38:45.889074 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:38:45.889039 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:39:34.297029 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:39:34.296988 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:39:46.588066 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:39:46.588032 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:40:30.320475 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.320380 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw"] Jul 28 22:40:30.320919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.320758 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7190a5a-56ba-4d6b-865b-8fe988b39f06" containerName="payload-pre-processing" Jul 28 22:40:30.320919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.320769 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7190a5a-56ba-4d6b-865b-8fe988b39f06" containerName="payload-pre-processing" Jul 28 22:40:30.320919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.320790 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80372bf4-853c-4b45-8cb9-b5b94d3ac0c4" containerName="payload-processing" Jul 28 22:40:30.320919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.320796 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="80372bf4-853c-4b45-8cb9-b5b94d3ac0c4" containerName="payload-processing" Jul 28 22:40:30.320919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.320860 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="80372bf4-853c-4b45-8cb9-b5b94d3ac0c4" containerName="payload-processing" Jul 28 22:40:30.320919 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.320873 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7190a5a-56ba-4d6b-865b-8fe988b39f06" containerName="payload-pre-processing" Jul 28 22:40:30.323879 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.323863 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:30.328003 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.327974 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-00957a\"" Jul 28 22:40:30.328003 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.327991 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-00957a-dockercfg-nsmcp\"" Jul 28 22:40:30.341667 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.341637 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw"] Jul 28 22:40:30.370285 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.370254 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19503c90-e0db-49e0-ba11-fe1768d5b7a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw\" (UID: \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:30.370450 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.370377 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tx8n\" (UniqueName: \"kubernetes.io/projected/19503c90-e0db-49e0-ba11-fe1768d5b7a8-kube-api-access-7tx8n\") pod \"payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw\" (UID: \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:30.389154 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.389117 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt"] Jul 28 22:40:30.393139 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.393117 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:40:30.403168 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.403144 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt"] Jul 28 22:40:30.471311 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.471280 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2mp4\" (UniqueName: \"kubernetes.io/projected/9344ae03-86d5-46c8-b732-2342ea89d257-kube-api-access-w2mp4\") pod \"payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt\" (UID: \"9344ae03-86d5-46c8-b732-2342ea89d257\") " pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:40:30.471508 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.471351 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tx8n\" (UniqueName: \"kubernetes.io/projected/19503c90-e0db-49e0-ba11-fe1768d5b7a8-kube-api-access-7tx8n\") pod \"payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw\" (UID: \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:30.471508 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.471393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9344ae03-86d5-46c8-b732-2342ea89d257-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt\" (UID: \"9344ae03-86d5-46c8-b732-2342ea89d257\") " pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:40:30.471508 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.471426 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19503c90-e0db-49e0-ba11-fe1768d5b7a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw\" (UID: \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:30.472091 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.472073 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19503c90-e0db-49e0-ba11-fe1768d5b7a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw\" (UID: \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:30.480361 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.480337 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tx8n\" (UniqueName: \"kubernetes.io/projected/19503c90-e0db-49e0-ba11-fe1768d5b7a8-kube-api-access-7tx8n\") pod \"payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw\" (UID: \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:30.572656 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.572563 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9344ae03-86d5-46c8-b732-2342ea89d257-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt\" (UID: \"9344ae03-86d5-46c8-b732-2342ea89d257\") " pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:40:30.572656 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.572634 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2mp4\" (UniqueName: \"kubernetes.io/projected/9344ae03-86d5-46c8-b732-2342ea89d257-kube-api-access-w2mp4\") pod \"payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt\" (UID: \"9344ae03-86d5-46c8-b732-2342ea89d257\") " pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:40:30.573328 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.573308 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9344ae03-86d5-46c8-b732-2342ea89d257-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt\" (UID: \"9344ae03-86d5-46c8-b732-2342ea89d257\") " pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:40:30.581310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.581286 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2mp4\" (UniqueName: \"kubernetes.io/projected/9344ae03-86d5-46c8-b732-2342ea89d257-kube-api-access-w2mp4\") pod \"payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt\" (UID: \"9344ae03-86d5-46c8-b732-2342ea89d257\") " pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:40:30.634147 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.634119 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:30.703911 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.703876 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:40:30.979430 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:30.979407 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw"] Jul 28 22:40:30.981509 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:40:30.981485 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19503c90_e0db_49e0_ba11_fe1768d5b7a8.slice/crio-f7aa8b5477e9eb24a507a48e922133e706abd466a308e6ef06adabbe821d7ed7 WatchSource:0}: Error finding container f7aa8b5477e9eb24a507a48e922133e706abd466a308e6ef06adabbe821d7ed7: Status 404 returned error can't find the container with id f7aa8b5477e9eb24a507a48e922133e706abd466a308e6ef06adabbe821d7ed7 Jul 28 22:40:31.053083 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:31.053057 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt"] Jul 28 22:40:31.055822 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:40:31.055797 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9344ae03_86d5_46c8_b732_2342ea89d257.slice/crio-4aceb2226247684f53781ec773bfe3269caed2ae60ef7b2a08be8ba93f5d85e0 WatchSource:0}: Error finding container 4aceb2226247684f53781ec773bfe3269caed2ae60ef7b2a08be8ba93f5d85e0: Status 404 returned error can't find the container with id 4aceb2226247684f53781ec773bfe3269caed2ae60ef7b2a08be8ba93f5d85e0 Jul 28 22:40:31.382360 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:31.382259 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" event={"ID":"19503c90-e0db-49e0-ba11-fe1768d5b7a8","Type":"ContainerStarted","Data":"f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538"} Jul 28 22:40:31.382360 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:31.382317 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" event={"ID":"19503c90-e0db-49e0-ba11-fe1768d5b7a8","Type":"ContainerStarted","Data":"f7aa8b5477e9eb24a507a48e922133e706abd466a308e6ef06adabbe821d7ed7"} Jul 28 22:40:31.382858 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:31.382371 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:31.384097 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:31.384073 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" event={"ID":"9344ae03-86d5-46c8-b732-2342ea89d257","Type":"ContainerStarted","Data":"97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88"} Jul 28 22:40:31.384221 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:31.384109 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" event={"ID":"9344ae03-86d5-46c8-b732-2342ea89d257","Type":"ContainerStarted","Data":"4aceb2226247684f53781ec773bfe3269caed2ae60ef7b2a08be8ba93f5d85e0"} Jul 28 22:40:31.384221 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:31.384138 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:40:31.414648 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:31.414598 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" podStartSLOduration=1.414581312 podStartE2EDuration="1.414581312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:40:31.414439234 +0000 UTC m=+2156.052298363" watchObservedRunningTime="2026-07-28 22:40:31.414581312 +0000 UTC m=+2156.052440440" Jul 28 22:40:31.415632 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:31.415603 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" podStartSLOduration=1.415591495 podStartE2EDuration="1.415591495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:40:31.398059729 +0000 UTC m=+2156.035918858" watchObservedRunningTime="2026-07-28 22:40:31.415591495 +0000 UTC m=+2156.053450625" Jul 28 22:40:35.797447 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:35.797414 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:40:42.391400 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:42.391369 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:40:42.391881 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:40:42.391441 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:41:01.668272 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.668235 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw"] Jul 28 22:41:01.672018 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.671999 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:01.676961 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.676915 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8a6fb9\"" Jul 28 22:41:01.677112 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.677039 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8a6fb9-dockercfg-qpldj\"" Jul 28 22:41:01.683563 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.683535 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw"] Jul 28 22:41:01.690844 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.690820 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s"] Jul 28 22:41:01.695194 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.695171 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:01.702681 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.702655 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s"] Jul 28 22:41:01.767443 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.767406 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee929db7-b34e-4435-b6e5-a8881d744de5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw\" (UID: \"ee929db7-b34e-4435-b6e5-a8881d744de5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:01.767625 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.767476 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5wk\" (UniqueName: \"kubernetes.io/projected/ee929db7-b34e-4435-b6e5-a8881d744de5-kube-api-access-4r5wk\") pod \"payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw\" (UID: \"ee929db7-b34e-4435-b6e5-a8881d744de5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:01.868347 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.868307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee929db7-b34e-4435-b6e5-a8881d744de5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw\" (UID: \"ee929db7-b34e-4435-b6e5-a8881d744de5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:01.868538 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.868371 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7n2\" (UniqueName: \"kubernetes.io/projected/783e7caf-afe2-406f-b6a5-74b287281139-kube-api-access-xr7n2\") pod \"payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s\" (UID: \"783e7caf-afe2-406f-b6a5-74b287281139\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:01.868538 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.868404 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5wk\" (UniqueName: \"kubernetes.io/projected/ee929db7-b34e-4435-b6e5-a8881d744de5-kube-api-access-4r5wk\") pod \"payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw\" (UID: \"ee929db7-b34e-4435-b6e5-a8881d744de5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:01.868538 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.868434 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783e7caf-afe2-406f-b6a5-74b287281139-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s\" (UID: \"783e7caf-afe2-406f-b6a5-74b287281139\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:01.869076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.869057 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee929db7-b34e-4435-b6e5-a8881d744de5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw\" (UID: \"ee929db7-b34e-4435-b6e5-a8881d744de5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:01.876711 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.876684 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5wk\" (UniqueName: \"kubernetes.io/projected/ee929db7-b34e-4435-b6e5-a8881d744de5-kube-api-access-4r5wk\") pod \"payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw\" (UID: \"ee929db7-b34e-4435-b6e5-a8881d744de5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:01.969412 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.969380 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783e7caf-afe2-406f-b6a5-74b287281139-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s\" (UID: \"783e7caf-afe2-406f-b6a5-74b287281139\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:01.969581 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.969477 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7n2\" (UniqueName: \"kubernetes.io/projected/783e7caf-afe2-406f-b6a5-74b287281139-kube-api-access-xr7n2\") pod \"payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s\" (UID: \"783e7caf-afe2-406f-b6a5-74b287281139\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:01.970208 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.970190 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783e7caf-afe2-406f-b6a5-74b287281139-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s\" (UID: \"783e7caf-afe2-406f-b6a5-74b287281139\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:01.981838 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.981813 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7n2\" (UniqueName: \"kubernetes.io/projected/783e7caf-afe2-406f-b6a5-74b287281139-kube-api-access-xr7n2\") pod \"payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s\" (UID: \"783e7caf-afe2-406f-b6a5-74b287281139\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:01.984683 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:01.984666 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:02.008069 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.008036 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:02.160051 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.160027 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s"] Jul 28 22:41:02.162005 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:41:02.161980 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783e7caf_afe2_406f_b6a5_74b287281139.slice/crio-3d72124b598e87a73c690142b5561f5171c48944be5a7741d3c3ce4ed73109eb WatchSource:0}: Error finding container 3d72124b598e87a73c690142b5561f5171c48944be5a7741d3c3ce4ed73109eb: Status 404 returned error can't find the container with id 3d72124b598e87a73c690142b5561f5171c48944be5a7741d3c3ce4ed73109eb Jul 28 22:41:02.338219 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.338193 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw"] Jul 28 22:41:02.339978 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:41:02.339888 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee929db7_b34e_4435_b6e5_a8881d744de5.slice/crio-6b98179a971db8973f490e2c939f664dd19ed350926afc105c9b39311915fbe5 WatchSource:0}: Error finding container 6b98179a971db8973f490e2c939f664dd19ed350926afc105c9b39311915fbe5: Status 404 returned error can't find the container with id 6b98179a971db8973f490e2c939f664dd19ed350926afc105c9b39311915fbe5 Jul 28 22:41:02.535248 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.535130 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" event={"ID":"783e7caf-afe2-406f-b6a5-74b287281139","Type":"ContainerStarted","Data":"cfd6a00bc1e03a63a613cde472ffdf92c1a3adf1c2850ddbce50cb68227cf34b"} Jul 28 22:41:02.535248 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.535186 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" event={"ID":"783e7caf-afe2-406f-b6a5-74b287281139","Type":"ContainerStarted","Data":"3d72124b598e87a73c690142b5561f5171c48944be5a7741d3c3ce4ed73109eb"} Jul 28 22:41:02.535248 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.535217 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:02.537169 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.537123 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" event={"ID":"ee929db7-b34e-4435-b6e5-a8881d744de5","Type":"ContainerStarted","Data":"084f55210213922a2242a08680082a432d453c27ffd49fdbd1f6f18753d26370"} Jul 28 22:41:02.537371 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.537201 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" event={"ID":"ee929db7-b34e-4435-b6e5-a8881d744de5","Type":"ContainerStarted","Data":"6b98179a971db8973f490e2c939f664dd19ed350926afc105c9b39311915fbe5"} Jul 28 22:41:02.537371 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.537230 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:02.550799 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.550753 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" podStartSLOduration=1.5507384690000001 podStartE2EDuration="1.550738469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:41:02.549813243 +0000 UTC m=+2187.187672398" watchObservedRunningTime="2026-07-28 22:41:02.550738469 +0000 UTC m=+2187.188597603" Jul 28 22:41:02.566088 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:02.566038 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" podStartSLOduration=1.566020991 podStartE2EDuration="1.566020991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:41:02.564237989 +0000 UTC m=+2187.202097121" watchObservedRunningTime="2026-07-28 22:41:02.566020991 +0000 UTC m=+2187.203880192" Jul 28 22:41:05.889676 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:05.889628 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:41:13.544084 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:13.544050 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:41:13.544745 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:13.544127 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:41:34.097088 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:34.097045 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:41:58.795446 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:41:58.795361 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:42:13.290781 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:42:13.290738 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:42:54.897773 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:42:54.897732 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:43:02.252633 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.252595 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt"] Jul 28 22:43:02.253132 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.252862 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" podUID="9344ae03-86d5-46c8-b732-2342ea89d257" containerName="payload-processing" containerID="cri-o://97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88" gracePeriod=30 Jul 28 22:43:02.263832 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.260882 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw"] Jul 28 22:43:02.263832 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.261131 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" podUID="19503c90-e0db-49e0-ba11-fe1768d5b7a8" containerName="payload-pre-processing" containerID="cri-o://f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538" gracePeriod=30 Jul 28 22:43:02.389202 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.389150 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" podUID="9344ae03-86d5-46c8-b732-2342ea89d257" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.45:9004: connect: connection refused" Jul 28 22:43:02.389369 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.389150 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" podUID="19503c90-e0db-49e0-ba11-fe1768d5b7a8" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.44:9004: connect: connection refused" Jul 28 22:43:02.528385 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.528343 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:43:02.548660 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.548629 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:43:02.682237 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.682205 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2mp4\" (UniqueName: \"kubernetes.io/projected/9344ae03-86d5-46c8-b732-2342ea89d257-kube-api-access-w2mp4\") pod \"9344ae03-86d5-46c8-b732-2342ea89d257\" (UID: \"9344ae03-86d5-46c8-b732-2342ea89d257\") " Jul 28 22:43:02.682404 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.682282 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19503c90-e0db-49e0-ba11-fe1768d5b7a8-plugins-config-volume\") pod \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\" (UID: \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\") " Jul 28 22:43:02.682404 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.682372 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tx8n\" (UniqueName: \"kubernetes.io/projected/19503c90-e0db-49e0-ba11-fe1768d5b7a8-kube-api-access-7tx8n\") pod \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\" (UID: \"19503c90-e0db-49e0-ba11-fe1768d5b7a8\") " Jul 28 22:43:02.682539 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.682463 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9344ae03-86d5-46c8-b732-2342ea89d257-plugins-config-volume\") pod \"9344ae03-86d5-46c8-b732-2342ea89d257\" (UID: \"9344ae03-86d5-46c8-b732-2342ea89d257\") " Jul 28 22:43:02.682699 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.682669 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19503c90-e0db-49e0-ba11-fe1768d5b7a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19503c90-e0db-49e0-ba11-fe1768d5b7a8" (UID: "19503c90-e0db-49e0-ba11-fe1768d5b7a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:43:02.682800 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.682781 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19503c90-e0db-49e0-ba11-fe1768d5b7a8-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:43:02.682851 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.682825 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9344ae03-86d5-46c8-b732-2342ea89d257-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9344ae03-86d5-46c8-b732-2342ea89d257" (UID: "9344ae03-86d5-46c8-b732-2342ea89d257"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:43:02.684473 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.684440 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9344ae03-86d5-46c8-b732-2342ea89d257-kube-api-access-w2mp4" (OuterVolumeSpecName: "kube-api-access-w2mp4") pod "9344ae03-86d5-46c8-b732-2342ea89d257" (UID: "9344ae03-86d5-46c8-b732-2342ea89d257"). InnerVolumeSpecName "kube-api-access-w2mp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:43:02.684787 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.684770 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19503c90-e0db-49e0-ba11-fe1768d5b7a8-kube-api-access-7tx8n" (OuterVolumeSpecName: "kube-api-access-7tx8n") pod "19503c90-e0db-49e0-ba11-fe1768d5b7a8" (UID: "19503c90-e0db-49e0-ba11-fe1768d5b7a8"). InnerVolumeSpecName "kube-api-access-7tx8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:43:02.783964 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.783839 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tx8n\" (UniqueName: \"kubernetes.io/projected/19503c90-e0db-49e0-ba11-fe1768d5b7a8-kube-api-access-7tx8n\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:43:02.783964 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.783890 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9344ae03-86d5-46c8-b732-2342ea89d257-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:43:02.783964 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:02.783899 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2mp4\" (UniqueName: \"kubernetes.io/projected/9344ae03-86d5-46c8-b732-2342ea89d257-kube-api-access-w2mp4\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:43:03.135655 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.135563 2557 generic.go:356] "Generic (PLEG): container finished" podID="19503c90-e0db-49e0-ba11-fe1768d5b7a8" containerID="f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538" exitCode=0 Jul 28 22:43:03.135655 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.135611 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" Jul 28 22:43:03.135874 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.135651 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" event={"ID":"19503c90-e0db-49e0-ba11-fe1768d5b7a8","Type":"ContainerDied","Data":"f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538"} Jul 28 22:43:03.135874 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.135709 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw" event={"ID":"19503c90-e0db-49e0-ba11-fe1768d5b7a8","Type":"ContainerDied","Data":"f7aa8b5477e9eb24a507a48e922133e706abd466a308e6ef06adabbe821d7ed7"} Jul 28 22:43:03.135874 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.135740 2557 scope.go:117] "RemoveContainer" containerID="f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538" Jul 28 22:43:03.137116 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.137086 2557 generic.go:356] "Generic (PLEG): container finished" podID="9344ae03-86d5-46c8-b732-2342ea89d257" containerID="97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88" exitCode=0 Jul 28 22:43:03.137232 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.137113 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" event={"ID":"9344ae03-86d5-46c8-b732-2342ea89d257","Type":"ContainerDied","Data":"97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88"} Jul 28 22:43:03.137232 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.137125 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" Jul 28 22:43:03.137232 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.137159 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt" event={"ID":"9344ae03-86d5-46c8-b732-2342ea89d257","Type":"ContainerDied","Data":"4aceb2226247684f53781ec773bfe3269caed2ae60ef7b2a08be8ba93f5d85e0"} Jul 28 22:43:03.145969 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.145919 2557 scope.go:117] "RemoveContainer" containerID="f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538" Jul 28 22:43:03.146320 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:43:03.146291 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538\": container with ID starting with f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538 not found: ID does not exist" containerID="f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538" Jul 28 22:43:03.146422 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.146332 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538"} err="failed to get container status \"f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538\": rpc error: code = NotFound desc = could not find container \"f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538\": container with ID starting with f65fe0fd7466c5fc30b627faa315db7de32b8d5983842be9f04b3e4ac2846538 not found: ID does not exist" Jul 28 22:43:03.146422 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.146353 2557 scope.go:117] "RemoveContainer" containerID="97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88" Jul 28 22:43:03.154601 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.154578 2557 scope.go:117] "RemoveContainer" containerID="97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88" Jul 28 22:43:03.154874 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:43:03.154855 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88\": container with ID starting with 97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88 not found: ID does not exist" containerID="97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88" Jul 28 22:43:03.154937 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.154883 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88"} err="failed to get container status \"97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88\": rpc error: code = NotFound desc = could not find container \"97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88\": container with ID starting with 97930e7eeb1f3ea123c2a6c389dbaf924cef8883ca2c38a178bf0985d06b5b88 not found: ID does not exist" Jul 28 22:43:03.161575 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.161544 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw"] Jul 28 22:43:03.165859 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.165835 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00957a-786d6c8c56-27rlw"] Jul 28 22:43:03.178977 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.178948 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt"] Jul 28 22:43:03.181804 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:03.181778 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-00957a-57bd85f8bb-qsxzt"] Jul 28 22:43:04.045430 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:04.045396 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19503c90-e0db-49e0-ba11-fe1768d5b7a8" path="/var/lib/kubelet/pods/19503c90-e0db-49e0-ba11-fe1768d5b7a8/volumes" Jul 28 22:43:04.045803 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:04.045753 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9344ae03-86d5-46c8-b732-2342ea89d257" path="/var/lib/kubelet/pods/9344ae03-86d5-46c8-b732-2342ea89d257/volumes" Jul 28 22:43:07.496085 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:07.496049 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:43:20.072606 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.072576 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s"] Jul 28 22:43:20.073081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.072826 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" podUID="783e7caf-afe2-406f-b6a5-74b287281139" containerName="payload-processing" containerID="cri-o://cfd6a00bc1e03a63a613cde472ffdf92c1a3adf1c2850ddbce50cb68227cf34b" gracePeriod=30 Jul 28 22:43:20.096209 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.096182 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw"] Jul 28 22:43:20.096421 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.096379 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" podUID="ee929db7-b34e-4435-b6e5-a8881d744de5" containerName="payload-pre-processing" containerID="cri-o://084f55210213922a2242a08680082a432d453c27ffd49fdbd1f6f18753d26370" gracePeriod=30 Jul 28 22:43:20.221846 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.221808 2557 generic.go:356] "Generic (PLEG): container finished" podID="783e7caf-afe2-406f-b6a5-74b287281139" containerID="cfd6a00bc1e03a63a613cde472ffdf92c1a3adf1c2850ddbce50cb68227cf34b" exitCode=0 Jul 28 22:43:20.222026 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.221883 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" event={"ID":"783e7caf-afe2-406f-b6a5-74b287281139","Type":"ContainerDied","Data":"cfd6a00bc1e03a63a613cde472ffdf92c1a3adf1c2850ddbce50cb68227cf34b"} Jul 28 22:43:20.223677 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.223652 2557 generic.go:356] "Generic (PLEG): container finished" podID="ee929db7-b34e-4435-b6e5-a8881d744de5" containerID="084f55210213922a2242a08680082a432d453c27ffd49fdbd1f6f18753d26370" exitCode=0 Jul 28 22:43:20.223769 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.223696 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" event={"ID":"ee929db7-b34e-4435-b6e5-a8881d744de5","Type":"ContainerDied","Data":"084f55210213922a2242a08680082a432d453c27ffd49fdbd1f6f18753d26370"} Jul 28 22:43:20.375410 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.375386 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:43:20.378241 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.378212 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:43:20.453171 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.453134 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r5wk\" (UniqueName: \"kubernetes.io/projected/ee929db7-b34e-4435-b6e5-a8881d744de5-kube-api-access-4r5wk\") pod \"ee929db7-b34e-4435-b6e5-a8881d744de5\" (UID: \"ee929db7-b34e-4435-b6e5-a8881d744de5\") " Jul 28 22:43:20.453372 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.453182 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee929db7-b34e-4435-b6e5-a8881d744de5-plugins-config-volume\") pod \"ee929db7-b34e-4435-b6e5-a8881d744de5\" (UID: \"ee929db7-b34e-4435-b6e5-a8881d744de5\") " Jul 28 22:43:20.453372 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.453200 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783e7caf-afe2-406f-b6a5-74b287281139-plugins-config-volume\") pod \"783e7caf-afe2-406f-b6a5-74b287281139\" (UID: \"783e7caf-afe2-406f-b6a5-74b287281139\") " Jul 28 22:43:20.453372 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.453276 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr7n2\" (UniqueName: \"kubernetes.io/projected/783e7caf-afe2-406f-b6a5-74b287281139-kube-api-access-xr7n2\") pod \"783e7caf-afe2-406f-b6a5-74b287281139\" (UID: \"783e7caf-afe2-406f-b6a5-74b287281139\") " Jul 28 22:43:20.453690 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.453664 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783e7caf-afe2-406f-b6a5-74b287281139-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "783e7caf-afe2-406f-b6a5-74b287281139" (UID: "783e7caf-afe2-406f-b6a5-74b287281139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:43:20.453760 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.453672 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee929db7-b34e-4435-b6e5-a8881d744de5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee929db7-b34e-4435-b6e5-a8881d744de5" (UID: "ee929db7-b34e-4435-b6e5-a8881d744de5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:43:20.455392 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.455371 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783e7caf-afe2-406f-b6a5-74b287281139-kube-api-access-xr7n2" (OuterVolumeSpecName: "kube-api-access-xr7n2") pod "783e7caf-afe2-406f-b6a5-74b287281139" (UID: "783e7caf-afe2-406f-b6a5-74b287281139"). InnerVolumeSpecName "kube-api-access-xr7n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:43:20.455496 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.455452 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee929db7-b34e-4435-b6e5-a8881d744de5-kube-api-access-4r5wk" (OuterVolumeSpecName: "kube-api-access-4r5wk") pod "ee929db7-b34e-4435-b6e5-a8881d744de5" (UID: "ee929db7-b34e-4435-b6e5-a8881d744de5"). InnerVolumeSpecName "kube-api-access-4r5wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:43:20.554503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.554468 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr7n2\" (UniqueName: \"kubernetes.io/projected/783e7caf-afe2-406f-b6a5-74b287281139-kube-api-access-xr7n2\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:43:20.554503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.554498 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5wk\" (UniqueName: \"kubernetes.io/projected/ee929db7-b34e-4435-b6e5-a8881d744de5-kube-api-access-4r5wk\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:43:20.554503 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.554507 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee929db7-b34e-4435-b6e5-a8881d744de5-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:43:20.554742 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:20.554516 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783e7caf-afe2-406f-b6a5-74b287281139-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:43:21.230070 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.230043 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" Jul 28 22:43:21.230506 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.230042 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s" event={"ID":"783e7caf-afe2-406f-b6a5-74b287281139","Type":"ContainerDied","Data":"3d72124b598e87a73c690142b5561f5171c48944be5a7741d3c3ce4ed73109eb"} Jul 28 22:43:21.230506 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.230169 2557 scope.go:117] "RemoveContainer" containerID="cfd6a00bc1e03a63a613cde472ffdf92c1a3adf1c2850ddbce50cb68227cf34b" Jul 28 22:43:21.231478 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.231458 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" Jul 28 22:43:21.231478 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.231457 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw" event={"ID":"ee929db7-b34e-4435-b6e5-a8881d744de5","Type":"ContainerDied","Data":"6b98179a971db8973f490e2c939f664dd19ed350926afc105c9b39311915fbe5"} Jul 28 22:43:21.239606 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.239579 2557 scope.go:117] "RemoveContainer" containerID="084f55210213922a2242a08680082a432d453c27ffd49fdbd1f6f18753d26370" Jul 28 22:43:21.254881 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.254855 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw"] Jul 28 22:43:21.259023 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.258998 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a6fb9-59d59f8bb4-4bjdw"] Jul 28 22:43:21.268770 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.268740 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s"] Jul 28 22:43:21.272591 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:21.272569 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a6fb9-7586685684-krr5s"] Jul 28 22:43:22.045642 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:22.045609 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783e7caf-afe2-406f-b6a5-74b287281139" path="/var/lib/kubelet/pods/783e7caf-afe2-406f-b6a5-74b287281139/volumes" Jul 28 22:43:22.046002 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:22.045988 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee929db7-b34e-4435-b6e5-a8881d744de5" path="/var/lib/kubelet/pods/ee929db7-b34e-4435-b6e5-a8881d744de5/volumes" Jul 28 22:43:22.898673 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:22.898641 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:43:31.067984 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.067892 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp"] Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068263 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9344ae03-86d5-46c8-b732-2342ea89d257" containerName="payload-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068274 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9344ae03-86d5-46c8-b732-2342ea89d257" containerName="payload-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068286 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19503c90-e0db-49e0-ba11-fe1768d5b7a8" containerName="payload-pre-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068292 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="19503c90-e0db-49e0-ba11-fe1768d5b7a8" containerName="payload-pre-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068313 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783e7caf-afe2-406f-b6a5-74b287281139" containerName="payload-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068320 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="783e7caf-afe2-406f-b6a5-74b287281139" containerName="payload-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068326 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee929db7-b34e-4435-b6e5-a8881d744de5" containerName="payload-pre-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068332 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee929db7-b34e-4435-b6e5-a8881d744de5" containerName="payload-pre-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068386 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee929db7-b34e-4435-b6e5-a8881d744de5" containerName="payload-pre-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068395 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="19503c90-e0db-49e0-ba11-fe1768d5b7a8" containerName="payload-pre-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068402 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9344ae03-86d5-46c8-b732-2342ea89d257" containerName="payload-processing" Jul 28 22:43:31.068442 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.068408 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="783e7caf-afe2-406f-b6a5-74b287281139" containerName="payload-processing" Jul 28 22:43:31.073181 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.073161 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:31.076050 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.076024 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-eeffd6-dockercfg-q5gxb\"" Jul 28 22:43:31.076331 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.076232 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-eeffd6\"" Jul 28 22:43:31.079814 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.079791 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp"] Jul 28 22:43:31.101739 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.101715 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn"] Jul 28 22:43:31.105267 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.105251 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:31.115389 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.115368 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn"] Jul 28 22:43:31.141112 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.141082 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4fe769-6f7c-45d8-a827-2be43576ceb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp\" (UID: \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:31.141268 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.141142 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcj7s\" (UniqueName: \"kubernetes.io/projected/2c4fe769-6f7c-45d8-a827-2be43576ceb1-kube-api-access-fcj7s\") pod \"payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp\" (UID: \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:31.141268 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.141223 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn\" (UID: \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\") " pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:31.141368 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.141285 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtbc\" (UniqueName: \"kubernetes.io/projected/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-kube-api-access-vrtbc\") pod \"payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn\" (UID: \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\") " pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:31.242676 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.242632 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn\" (UID: \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\") " pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:31.242853 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.242698 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtbc\" (UniqueName: \"kubernetes.io/projected/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-kube-api-access-vrtbc\") pod \"payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn\" (UID: \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\") " pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:31.242853 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.242833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4fe769-6f7c-45d8-a827-2be43576ceb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp\" (UID: \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:31.242965 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.242913 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcj7s\" (UniqueName: \"kubernetes.io/projected/2c4fe769-6f7c-45d8-a827-2be43576ceb1-kube-api-access-fcj7s\") pod \"payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp\" (UID: \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:31.243380 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.243364 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn\" (UID: \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\") " pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:31.243511 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.243488 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4fe769-6f7c-45d8-a827-2be43576ceb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp\" (UID: \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:31.250836 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.250815 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtbc\" (UniqueName: \"kubernetes.io/projected/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-kube-api-access-vrtbc\") pod \"payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn\" (UID: \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\") " pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:31.250970 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.250912 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcj7s\" (UniqueName: \"kubernetes.io/projected/2c4fe769-6f7c-45d8-a827-2be43576ceb1-kube-api-access-fcj7s\") pod \"payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp\" (UID: \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:31.384986 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.384873 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:31.415751 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.415724 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:31.533705 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.533679 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp"] Jul 28 22:43:31.535661 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:43:31.535630 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4fe769_6f7c_45d8_a827_2be43576ceb1.slice/crio-427383988ae9adf44d5d4031375ab44e41a9a816266310e9c42042f04adc0a3d WatchSource:0}: Error finding container 427383988ae9adf44d5d4031375ab44e41a9a816266310e9c42042f04adc0a3d: Status 404 returned error can't find the container with id 427383988ae9adf44d5d4031375ab44e41a9a816266310e9c42042f04adc0a3d Jul 28 22:43:31.568311 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:31.568284 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn"] Jul 28 22:43:31.571409 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:43:31.571387 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a797ce_4d77_4b21_b24c_e02a2a8f4e53.slice/crio-53be895a29f84f5dc735dba496d5b255d2cd69c6cd1bdf90f58e3017aebb5b19 WatchSource:0}: Error finding container 53be895a29f84f5dc735dba496d5b255d2cd69c6cd1bdf90f58e3017aebb5b19: Status 404 returned error can't find the container with id 53be895a29f84f5dc735dba496d5b255d2cd69c6cd1bdf90f58e3017aebb5b19 Jul 28 22:43:32.287136 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:32.287099 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" event={"ID":"2c4fe769-6f7c-45d8-a827-2be43576ceb1","Type":"ContainerStarted","Data":"fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d"} Jul 28 22:43:32.287553 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:32.287162 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" event={"ID":"2c4fe769-6f7c-45d8-a827-2be43576ceb1","Type":"ContainerStarted","Data":"427383988ae9adf44d5d4031375ab44e41a9a816266310e9c42042f04adc0a3d"} Jul 28 22:43:32.287553 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:32.287198 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:32.288655 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:32.288628 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" event={"ID":"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53","Type":"ContainerStarted","Data":"d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362"} Jul 28 22:43:32.288762 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:32.288677 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" event={"ID":"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53","Type":"ContainerStarted","Data":"53be895a29f84f5dc735dba496d5b255d2cd69c6cd1bdf90f58e3017aebb5b19"} Jul 28 22:43:32.288762 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:32.288701 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:32.304088 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:32.304040 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" podStartSLOduration=1.304024558 podStartE2EDuration="1.304024558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:43:32.302869388 +0000 UTC m=+2336.940728518" watchObservedRunningTime="2026-07-28 22:43:32.304024558 +0000 UTC m=+2336.941883897" Jul 28 22:43:32.320351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:32.320298 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" podStartSLOduration=1.320284456 podStartE2EDuration="1.320284456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:43:32.318493415 +0000 UTC m=+2336.956352544" watchObservedRunningTime="2026-07-28 22:43:32.320284456 +0000 UTC m=+2336.958143584" Jul 28 22:43:34.797485 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:34.797443 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:43:43.295895 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:43.295865 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:43:43.296305 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:43.295967 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:43:59.624707 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.624669 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp"] Jul 28 22:43:59.628296 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.628281 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:43:59.630860 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.630836 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-57f127-dockercfg-kjrb4\"" Jul 28 22:43:59.630860 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.630850 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-57f127\"" Jul 28 22:43:59.638042 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.638020 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp"] Jul 28 22:43:59.661261 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.661234 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g"] Jul 28 22:43:59.664833 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.664813 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:43:59.676873 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.676848 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g"] Jul 28 22:43:59.812818 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.812774 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbl6k\" (UniqueName: \"kubernetes.io/projected/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-kube-api-access-cbl6k\") pod \"payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp\" (UID: \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:43:59.812818 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.812823 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzt6b\" (UniqueName: \"kubernetes.io/projected/a90d627c-8e4b-41cc-b0ea-885acbb678e3-kube-api-access-nzt6b\") pod \"payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g\" (UID: \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:43:59.813116 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.812847 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90d627c-8e4b-41cc-b0ea-885acbb678e3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g\" (UID: \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:43:59.813116 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.812976 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp\" (UID: \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:43:59.914024 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.913921 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90d627c-8e4b-41cc-b0ea-885acbb678e3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g\" (UID: \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:43:59.914024 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.913985 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp\" (UID: \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:43:59.914283 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.914069 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbl6k\" (UniqueName: \"kubernetes.io/projected/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-kube-api-access-cbl6k\") pod \"payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp\" (UID: \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:43:59.914283 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.914091 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzt6b\" (UniqueName: \"kubernetes.io/projected/a90d627c-8e4b-41cc-b0ea-885acbb678e3-kube-api-access-nzt6b\") pod \"payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g\" (UID: \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:43:59.914718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.914694 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90d627c-8e4b-41cc-b0ea-885acbb678e3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g\" (UID: \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:43:59.914797 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.914738 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp\" (UID: \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:43:59.922099 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.922068 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbl6k\" (UniqueName: \"kubernetes.io/projected/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-kube-api-access-cbl6k\") pod \"payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp\" (UID: \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:43:59.922099 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.922070 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzt6b\" (UniqueName: \"kubernetes.io/projected/a90d627c-8e4b-41cc-b0ea-885acbb678e3-kube-api-access-nzt6b\") pod \"payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g\" (UID: \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:43:59.938854 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.938828 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:43:59.975700 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:43:59.975664 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:44:00.105703 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.105676 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp"] Jul 28 22:44:00.107491 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:44:00.107463 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02bb144_dea1_4041_b5e1_e24cf5a0f7d8.slice/crio-634679d751619303b24caffe91e529f7fc779008df77332fac74469ba5e2510c WatchSource:0}: Error finding container 634679d751619303b24caffe91e529f7fc779008df77332fac74469ba5e2510c: Status 404 returned error can't find the container with id 634679d751619303b24caffe91e529f7fc779008df77332fac74469ba5e2510c Jul 28 22:44:00.145591 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.145563 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g"] Jul 28 22:44:00.147109 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:44:00.147084 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90d627c_8e4b_41cc_b0ea_885acbb678e3.slice/crio-066e1d128adad6fe6c1c0b7bc4896fb71d376d53c5f105ce366c820b00511fc2 WatchSource:0}: Error finding container 066e1d128adad6fe6c1c0b7bc4896fb71d376d53c5f105ce366c820b00511fc2: Status 404 returned error can't find the container with id 066e1d128adad6fe6c1c0b7bc4896fb71d376d53c5f105ce366c820b00511fc2 Jul 28 22:44:00.424261 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.424168 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" event={"ID":"a90d627c-8e4b-41cc-b0ea-885acbb678e3","Type":"ContainerStarted","Data":"371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08"} Jul 28 22:44:00.424261 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.424219 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" event={"ID":"a90d627c-8e4b-41cc-b0ea-885acbb678e3","Type":"ContainerStarted","Data":"066e1d128adad6fe6c1c0b7bc4896fb71d376d53c5f105ce366c820b00511fc2"} Jul 28 22:44:00.424487 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.424328 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:44:00.425678 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.425658 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" event={"ID":"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8","Type":"ContainerStarted","Data":"d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793"} Jul 28 22:44:00.425748 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.425697 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" event={"ID":"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8","Type":"ContainerStarted","Data":"634679d751619303b24caffe91e529f7fc779008df77332fac74469ba5e2510c"} Jul 28 22:44:00.425782 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.425746 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:44:00.440086 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.440041 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" podStartSLOduration=1.440026461 podStartE2EDuration="1.440026461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:00.438853132 +0000 UTC m=+2365.076712263" watchObservedRunningTime="2026-07-28 22:44:00.440026461 +0000 UTC m=+2365.077885590" Jul 28 22:44:00.456846 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:00.456798 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" podStartSLOduration=1.456781286 podStartE2EDuration="1.456781286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:00.456294052 +0000 UTC m=+2365.094153205" watchObservedRunningTime="2026-07-28 22:44:00.456781286 +0000 UTC m=+2365.094640416" Jul 28 22:44:07.797349 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:07.797309 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:44:11.432542 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:11.432511 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:44:11.433050 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:11.432599 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:44:42.868987 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:42.868952 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn"] Jul 28 22:44:42.869567 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:42.869208 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" podUID="c3a797ce-4d77-4b21-b24c-e02a2a8f4e53" containerName="payload-processing" containerID="cri-o://d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362" gracePeriod=30 Jul 28 22:44:42.875557 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:42.875520 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp"] Jul 28 22:44:42.875797 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:42.875735 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" podUID="2c4fe769-6f7c-45d8-a827-2be43576ceb1" containerName="payload-pre-processing" containerID="cri-o://fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d" gracePeriod=30 Jul 28 22:44:43.149719 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.149692 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:44:43.152455 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.152436 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:44:43.315355 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.315323 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-plugins-config-volume\") pod \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\" (UID: \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\") " Jul 28 22:44:43.315514 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.315365 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4fe769-6f7c-45d8-a827-2be43576ceb1-plugins-config-volume\") pod \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\" (UID: \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\") " Jul 28 22:44:43.315514 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.315386 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcj7s\" (UniqueName: \"kubernetes.io/projected/2c4fe769-6f7c-45d8-a827-2be43576ceb1-kube-api-access-fcj7s\") pod \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\" (UID: \"2c4fe769-6f7c-45d8-a827-2be43576ceb1\") " Jul 28 22:44:43.315514 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.315426 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrtbc\" (UniqueName: \"kubernetes.io/projected/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-kube-api-access-vrtbc\") pod \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\" (UID: \"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53\") " Jul 28 22:44:43.315688 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.315666 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3a797ce-4d77-4b21-b24c-e02a2a8f4e53" (UID: "c3a797ce-4d77-4b21-b24c-e02a2a8f4e53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:44:43.315753 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.315695 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4fe769-6f7c-45d8-a827-2be43576ceb1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c4fe769-6f7c-45d8-a827-2be43576ceb1" (UID: "2c4fe769-6f7c-45d8-a827-2be43576ceb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:44:43.317475 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.317451 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-kube-api-access-vrtbc" (OuterVolumeSpecName: "kube-api-access-vrtbc") pod "c3a797ce-4d77-4b21-b24c-e02a2a8f4e53" (UID: "c3a797ce-4d77-4b21-b24c-e02a2a8f4e53"). InnerVolumeSpecName "kube-api-access-vrtbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:44:43.317572 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.317529 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4fe769-6f7c-45d8-a827-2be43576ceb1-kube-api-access-fcj7s" (OuterVolumeSpecName: "kube-api-access-fcj7s") pod "2c4fe769-6f7c-45d8-a827-2be43576ceb1" (UID: "2c4fe769-6f7c-45d8-a827-2be43576ceb1"). InnerVolumeSpecName "kube-api-access-fcj7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:44:43.416398 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.416336 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4fe769-6f7c-45d8-a827-2be43576ceb1-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:44:43.416398 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.416357 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcj7s\" (UniqueName: \"kubernetes.io/projected/2c4fe769-6f7c-45d8-a827-2be43576ceb1-kube-api-access-fcj7s\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:44:43.416398 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.416369 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrtbc\" (UniqueName: \"kubernetes.io/projected/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-kube-api-access-vrtbc\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:44:43.416398 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.416377 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:44:43.640826 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.640790 2557 generic.go:356] "Generic (PLEG): container finished" podID="c3a797ce-4d77-4b21-b24c-e02a2a8f4e53" containerID="d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362" exitCode=0 Jul 28 22:44:43.641012 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.640833 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" Jul 28 22:44:43.641012 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.640877 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" event={"ID":"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53","Type":"ContainerDied","Data":"d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362"} Jul 28 22:44:43.641012 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.640945 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn" event={"ID":"c3a797ce-4d77-4b21-b24c-e02a2a8f4e53","Type":"ContainerDied","Data":"53be895a29f84f5dc735dba496d5b255d2cd69c6cd1bdf90f58e3017aebb5b19"} Jul 28 22:44:43.641012 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.640978 2557 scope.go:117] "RemoveContainer" containerID="d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362" Jul 28 22:44:43.642436 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.642413 2557 generic.go:356] "Generic (PLEG): container finished" podID="2c4fe769-6f7c-45d8-a827-2be43576ceb1" containerID="fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d" exitCode=0 Jul 28 22:44:43.642540 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.642454 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" Jul 28 22:44:43.642540 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.642453 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" event={"ID":"2c4fe769-6f7c-45d8-a827-2be43576ceb1","Type":"ContainerDied","Data":"fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d"} Jul 28 22:44:43.642540 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.642500 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp" event={"ID":"2c4fe769-6f7c-45d8-a827-2be43576ceb1","Type":"ContainerDied","Data":"427383988ae9adf44d5d4031375ab44e41a9a816266310e9c42042f04adc0a3d"} Jul 28 22:44:43.651295 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.651277 2557 scope.go:117] "RemoveContainer" containerID="d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362" Jul 28 22:44:43.651562 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:44:43.651545 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362\": container with ID starting with d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362 not found: ID does not exist" containerID="d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362" Jul 28 22:44:43.651609 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.651571 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362"} err="failed to get container status \"d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362\": rpc error: code = NotFound desc = could not find container \"d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362\": container with ID starting with d18a9f9ca1cb582c7b5238a956dbcb73c8cbb4139c7ea516555b357426f48362 not found: ID does not exist" Jul 28 22:44:43.651609 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.651588 2557 scope.go:117] "RemoveContainer" containerID="fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d" Jul 28 22:44:43.660235 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.659738 2557 scope.go:117] "RemoveContainer" containerID="fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d" Jul 28 22:44:43.660235 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:44:43.660031 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d\": container with ID starting with fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d not found: ID does not exist" containerID="fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d" Jul 28 22:44:43.660235 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.660059 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d"} err="failed to get container status \"fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d\": rpc error: code = NotFound desc = could not find container \"fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d\": container with ID starting with fdb47cb05b43dec1937e5fa85240629a9e632cbecfb897412a7d26ccc89a182d not found: ID does not exist" Jul 28 22:44:43.662345 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.662322 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn"] Jul 28 22:44:43.664016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.663988 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eeffd6-696f568c56-6trmn"] Jul 28 22:44:43.674406 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.674348 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp"] Jul 28 22:44:43.677043 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:43.677023 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eeffd6-7fd976ddb4-2g6xp"] Jul 28 22:44:44.045232 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:44.045195 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c4fe769-6f7c-45d8-a827-2be43576ceb1" path="/var/lib/kubelet/pods/2c4fe769-6f7c-45d8-a827-2be43576ceb1/volumes" Jul 28 22:44:44.045608 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:44.045573 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a797ce-4d77-4b21-b24c-e02a2a8f4e53" path="/var/lib/kubelet/pods/c3a797ce-4d77-4b21-b24c-e02a2a8f4e53/volumes" Jul 28 22:44:48.892011 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:44:48.891972 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:45:05.318587 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.318494 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g"] Jul 28 22:45:05.319060 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.318762 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" podUID="a90d627c-8e4b-41cc-b0ea-885acbb678e3" containerName="payload-processing" containerID="cri-o://371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08" gracePeriod=30 Jul 28 22:45:05.326903 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.326659 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp"] Jul 28 22:45:05.327175 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.327082 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" podUID="e02bb144-dea1-4041-b5e1-e24cf5a0f7d8" containerName="payload-pre-processing" containerID="cri-o://d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793" gracePeriod=30 Jul 28 22:45:05.605470 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.605446 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:45:05.608076 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.608059 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:45:05.719316 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.719271 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzt6b\" (UniqueName: \"kubernetes.io/projected/a90d627c-8e4b-41cc-b0ea-885acbb678e3-kube-api-access-nzt6b\") pod \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\" (UID: \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\") " Jul 28 22:45:05.719316 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.719331 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-plugins-config-volume\") pod \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\" (UID: \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\") " Jul 28 22:45:05.719568 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.719357 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbl6k\" (UniqueName: \"kubernetes.io/projected/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-kube-api-access-cbl6k\") pod \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\" (UID: \"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8\") " Jul 28 22:45:05.719568 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.719376 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90d627c-8e4b-41cc-b0ea-885acbb678e3-plugins-config-volume\") pod \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\" (UID: \"a90d627c-8e4b-41cc-b0ea-885acbb678e3\") " Jul 28 22:45:05.719787 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.719761 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e02bb144-dea1-4041-b5e1-e24cf5a0f7d8" (UID: "e02bb144-dea1-4041-b5e1-e24cf5a0f7d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:45:05.719872 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.719798 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90d627c-8e4b-41cc-b0ea-885acbb678e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a90d627c-8e4b-41cc-b0ea-885acbb678e3" (UID: "a90d627c-8e4b-41cc-b0ea-885acbb678e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:45:05.721434 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.721415 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-kube-api-access-cbl6k" (OuterVolumeSpecName: "kube-api-access-cbl6k") pod "e02bb144-dea1-4041-b5e1-e24cf5a0f7d8" (UID: "e02bb144-dea1-4041-b5e1-e24cf5a0f7d8"). InnerVolumeSpecName "kube-api-access-cbl6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:45:05.721481 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.721442 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90d627c-8e4b-41cc-b0ea-885acbb678e3-kube-api-access-nzt6b" (OuterVolumeSpecName: "kube-api-access-nzt6b") pod "a90d627c-8e4b-41cc-b0ea-885acbb678e3" (UID: "a90d627c-8e4b-41cc-b0ea-885acbb678e3"). InnerVolumeSpecName "kube-api-access-nzt6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:45:05.752148 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.752116 2557 generic.go:356] "Generic (PLEG): container finished" podID="a90d627c-8e4b-41cc-b0ea-885acbb678e3" containerID="371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08" exitCode=0 Jul 28 22:45:05.752310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.752167 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" Jul 28 22:45:05.752310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.752205 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" event={"ID":"a90d627c-8e4b-41cc-b0ea-885acbb678e3","Type":"ContainerDied","Data":"371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08"} Jul 28 22:45:05.752310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.752264 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g" event={"ID":"a90d627c-8e4b-41cc-b0ea-885acbb678e3","Type":"ContainerDied","Data":"066e1d128adad6fe6c1c0b7bc4896fb71d376d53c5f105ce366c820b00511fc2"} Jul 28 22:45:05.752310 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.752297 2557 scope.go:117] "RemoveContainer" containerID="371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08" Jul 28 22:45:05.753425 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.753404 2557 generic.go:356] "Generic (PLEG): container finished" podID="e02bb144-dea1-4041-b5e1-e24cf5a0f7d8" containerID="d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793" exitCode=0 Jul 28 22:45:05.753490 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.753442 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" Jul 28 22:45:05.753538 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.753443 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" event={"ID":"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8","Type":"ContainerDied","Data":"d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793"} Jul 28 22:45:05.753578 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.753547 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp" event={"ID":"e02bb144-dea1-4041-b5e1-e24cf5a0f7d8","Type":"ContainerDied","Data":"634679d751619303b24caffe91e529f7fc779008df77332fac74469ba5e2510c"} Jul 28 22:45:05.760887 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.760868 2557 scope.go:117] "RemoveContainer" containerID="371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08" Jul 28 22:45:05.761180 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:45:05.761159 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08\": container with ID starting with 371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08 not found: ID does not exist" containerID="371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08" Jul 28 22:45:05.761251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.761187 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08"} err="failed to get container status \"371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08\": rpc error: code = NotFound desc = could not find container \"371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08\": container with ID starting with 371cf1d3358f1884c9d6330e64bfd530fdd7ff5b4201d88c07d234b4ba784b08 not found: ID does not exist" Jul 28 22:45:05.761251 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.761203 2557 scope.go:117] "RemoveContainer" containerID="d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793" Jul 28 22:45:05.769204 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.769187 2557 scope.go:117] "RemoveContainer" containerID="d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793" Jul 28 22:45:05.769441 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:45:05.769423 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793\": container with ID starting with d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793 not found: ID does not exist" containerID="d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793" Jul 28 22:45:05.769502 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.769446 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793"} err="failed to get container status \"d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793\": rpc error: code = NotFound desc = could not find container \"d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793\": container with ID starting with d8db36521464b390204ad9e694b340c453325a84bc73a003e7cae406d8b3a793 not found: ID does not exist" Jul 28 22:45:05.777317 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.777295 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp"] Jul 28 22:45:05.782083 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.782065 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57f127-8548bf96d8-bg6kp"] Jul 28 22:45:05.791617 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.791591 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g"] Jul 28 22:45:05.795274 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.795256 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57f127-7669c7b9b6-6mc4g"] Jul 28 22:45:05.820795 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.820773 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzt6b\" (UniqueName: \"kubernetes.io/projected/a90d627c-8e4b-41cc-b0ea-885acbb678e3-kube-api-access-nzt6b\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:45:05.820795 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.820794 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:45:05.820948 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.820805 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbl6k\" (UniqueName: \"kubernetes.io/projected/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8-kube-api-access-cbl6k\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:45:05.820948 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:05.820814 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90d627c-8e4b-41cc-b0ea-885acbb678e3-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:45:06.045535 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:06.045505 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90d627c-8e4b-41cc-b0ea-885acbb678e3" path="/var/lib/kubelet/pods/a90d627c-8e4b-41cc-b0ea-885acbb678e3/volumes" Jul 28 22:45:06.045859 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:06.045846 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e02bb144-dea1-4041-b5e1-e24cf5a0f7d8" path="/var/lib/kubelet/pods/e02bb144-dea1-4041-b5e1-e24cf5a0f7d8/volumes" Jul 28 22:45:10.005876 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:10.005843 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:45:16.785536 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.785499 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m"] Jul 28 22:45:16.786049 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.785908 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c4fe769-6f7c-45d8-a827-2be43576ceb1" containerName="payload-pre-processing" Jul 28 22:45:16.786049 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.785935 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c4fe769-6f7c-45d8-a827-2be43576ceb1" containerName="payload-pre-processing" Jul 28 22:45:16.786049 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.785952 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90d627c-8e4b-41cc-b0ea-885acbb678e3" containerName="payload-processing" Jul 28 22:45:16.786049 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.785958 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90d627c-8e4b-41cc-b0ea-885acbb678e3" containerName="payload-processing" Jul 28 22:45:16.786049 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.785969 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a797ce-4d77-4b21-b24c-e02a2a8f4e53" containerName="payload-processing" Jul 28 22:45:16.786049 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.785975 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a797ce-4d77-4b21-b24c-e02a2a8f4e53" containerName="payload-processing" Jul 28 22:45:16.786049 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.785986 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e02bb144-dea1-4041-b5e1-e24cf5a0f7d8" containerName="payload-pre-processing" Jul 28 22:45:16.786049 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.785991 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e02bb144-dea1-4041-b5e1-e24cf5a0f7d8" containerName="payload-pre-processing" Jul 28 22:45:16.786306 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.786064 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90d627c-8e4b-41cc-b0ea-885acbb678e3" containerName="payload-processing" Jul 28 22:45:16.786306 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.786074 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e02bb144-dea1-4041-b5e1-e24cf5a0f7d8" containerName="payload-pre-processing" Jul 28 22:45:16.786306 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.786080 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c4fe769-6f7c-45d8-a827-2be43576ceb1" containerName="payload-pre-processing" Jul 28 22:45:16.786306 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.786090 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a797ce-4d77-4b21-b24c-e02a2a8f4e53" containerName="payload-processing" Jul 28 22:45:16.789180 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.789163 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:16.793629 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.793606 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-ae4fa9\"" Jul 28 22:45:16.794783 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.794757 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-ae4fa9-dockercfg-r58zl\"" Jul 28 22:45:16.807186 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.807165 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m"] Jul 28 22:45:16.824675 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.824648 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m"] Jul 28 22:45:16.828734 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.828715 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:16.837240 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.837215 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m"] Jul 28 22:45:16.918514 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.918479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5d913f-599e-4f98-9dc9-4b63a036a851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m\" (UID: \"ae5d913f-599e-4f98-9dc9-4b63a036a851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:16.918714 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.918522 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bvb8\" (UniqueName: \"kubernetes.io/projected/ae5d913f-599e-4f98-9dc9-4b63a036a851-kube-api-access-2bvb8\") pod \"payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m\" (UID: \"ae5d913f-599e-4f98-9dc9-4b63a036a851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:16.918714 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.918618 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gdc\" (UniqueName: \"kubernetes.io/projected/f18d3719-7633-4a89-ae74-d59abb0177c1-kube-api-access-z5gdc\") pod \"payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m\" (UID: \"f18d3719-7633-4a89-ae74-d59abb0177c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:16.918714 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:16.918666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18d3719-7633-4a89-ae74-d59abb0177c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m\" (UID: \"f18d3719-7633-4a89-ae74-d59abb0177c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:17.019466 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.019424 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gdc\" (UniqueName: \"kubernetes.io/projected/f18d3719-7633-4a89-ae74-d59abb0177c1-kube-api-access-z5gdc\") pod \"payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m\" (UID: \"f18d3719-7633-4a89-ae74-d59abb0177c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:17.019466 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.019472 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18d3719-7633-4a89-ae74-d59abb0177c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m\" (UID: \"f18d3719-7633-4a89-ae74-d59abb0177c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:17.019734 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.019522 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5d913f-599e-4f98-9dc9-4b63a036a851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m\" (UID: \"ae5d913f-599e-4f98-9dc9-4b63a036a851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:17.019734 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.019544 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bvb8\" (UniqueName: \"kubernetes.io/projected/ae5d913f-599e-4f98-9dc9-4b63a036a851-kube-api-access-2bvb8\") pod \"payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m\" (UID: \"ae5d913f-599e-4f98-9dc9-4b63a036a851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:17.020506 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.020483 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18d3719-7633-4a89-ae74-d59abb0177c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m\" (UID: \"f18d3719-7633-4a89-ae74-d59abb0177c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:17.020576 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.020483 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5d913f-599e-4f98-9dc9-4b63a036a851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m\" (UID: \"ae5d913f-599e-4f98-9dc9-4b63a036a851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:17.028099 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.028070 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gdc\" (UniqueName: \"kubernetes.io/projected/f18d3719-7633-4a89-ae74-d59abb0177c1-kube-api-access-z5gdc\") pod \"payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m\" (UID: \"f18d3719-7633-4a89-ae74-d59abb0177c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:17.028220 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.028165 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bvb8\" (UniqueName: \"kubernetes.io/projected/ae5d913f-599e-4f98-9dc9-4b63a036a851-kube-api-access-2bvb8\") pod \"payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m\" (UID: \"ae5d913f-599e-4f98-9dc9-4b63a036a851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:17.099533 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.099425 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:17.140199 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.140165 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:17.240939 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.240883 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m"] Jul 28 22:45:17.245824 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:45:17.245797 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18d3719_7633_4a89_ae74_d59abb0177c1.slice/crio-b1c2eb4f7cb84ebfd02354bf9977a16d67666299ae1d3384cb154826711b680f WatchSource:0}: Error finding container b1c2eb4f7cb84ebfd02354bf9977a16d67666299ae1d3384cb154826711b680f: Status 404 returned error can't find the container with id b1c2eb4f7cb84ebfd02354bf9977a16d67666299ae1d3384cb154826711b680f Jul 28 22:45:17.292544 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.292518 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m"] Jul 28 22:45:17.295130 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:45:17.295105 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5d913f_599e_4f98_9dc9_4b63a036a851.slice/crio-55b00ce0ed0caa1f41e464488dd27841201616edef6855bf5e3fb4feb69df163 WatchSource:0}: Error finding container 55b00ce0ed0caa1f41e464488dd27841201616edef6855bf5e3fb4feb69df163: Status 404 returned error can't find the container with id 55b00ce0ed0caa1f41e464488dd27841201616edef6855bf5e3fb4feb69df163 Jul 28 22:45:17.812372 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.812338 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" event={"ID":"f18d3719-7633-4a89-ae74-d59abb0177c1","Type":"ContainerStarted","Data":"a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6"} Jul 28 22:45:17.812808 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.812388 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" event={"ID":"f18d3719-7633-4a89-ae74-d59abb0177c1","Type":"ContainerStarted","Data":"b1c2eb4f7cb84ebfd02354bf9977a16d67666299ae1d3384cb154826711b680f"} Jul 28 22:45:17.812808 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.812430 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:17.813833 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.813813 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" event={"ID":"ae5d913f-599e-4f98-9dc9-4b63a036a851","Type":"ContainerStarted","Data":"0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114"} Jul 28 22:45:17.813945 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.813846 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" event={"ID":"ae5d913f-599e-4f98-9dc9-4b63a036a851","Type":"ContainerStarted","Data":"55b00ce0ed0caa1f41e464488dd27841201616edef6855bf5e3fb4feb69df163"} Jul 28 22:45:17.813945 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.813889 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:17.829297 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.829251 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" podStartSLOduration=1.829235026 podStartE2EDuration="1.829235026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:45:17.827579862 +0000 UTC m=+2442.465439018" watchObservedRunningTime="2026-07-28 22:45:17.829235026 +0000 UTC m=+2442.467094132" Jul 28 22:45:17.844134 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:17.844067 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" podStartSLOduration=1.844052317 podStartE2EDuration="1.844052317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:45:17.843350797 +0000 UTC m=+2442.481209928" watchObservedRunningTime="2026-07-28 22:45:17.844052317 +0000 UTC m=+2442.481911445" Jul 28 22:45:21.492474 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:21.492432 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:45:28.821144 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:28.821110 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:45:28.823463 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:28.821186 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:45:45.351480 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.351444 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz"] Jul 28 22:45:45.355264 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.355244 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:45:45.357964 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.357939 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-1661d9\"" Jul 28 22:45:45.366514 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.366493 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz"] Jul 28 22:45:45.383717 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.383675 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5"] Jul 28 22:45:45.388349 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.388326 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:45.391553 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.391532 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-1661d9-dockercfg-59nvw\"" Jul 28 22:45:45.392856 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.392819 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5"] Jul 28 22:45:45.486596 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.486561 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz\" (UID: \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:45:45.486776 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.486609 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgwxd\" (UniqueName: \"kubernetes.io/projected/a8aa3438-7d41-42c7-b7df-cd5792016f96-kube-api-access-dgwxd\") pod \"payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5\" (UID: \"a8aa3438-7d41-42c7-b7df-cd5792016f96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:45.486776 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.486733 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa3438-7d41-42c7-b7df-cd5792016f96-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5\" (UID: \"a8aa3438-7d41-42c7-b7df-cd5792016f96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:45.486865 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.486783 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krs45\" (UniqueName: \"kubernetes.io/projected/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-kube-api-access-krs45\") pod \"payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz\" (UID: \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:45:45.587595 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.587550 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa3438-7d41-42c7-b7df-cd5792016f96-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5\" (UID: \"a8aa3438-7d41-42c7-b7df-cd5792016f96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:45.587770 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.587605 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krs45\" (UniqueName: \"kubernetes.io/projected/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-kube-api-access-krs45\") pod \"payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz\" (UID: \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:45:45.587770 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.587652 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz\" (UID: \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:45:45.587770 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.587670 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgwxd\" (UniqueName: \"kubernetes.io/projected/a8aa3438-7d41-42c7-b7df-cd5792016f96-kube-api-access-dgwxd\") pod \"payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5\" (UID: \"a8aa3438-7d41-42c7-b7df-cd5792016f96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:45.588418 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.588399 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz\" (UID: \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:45:45.588470 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.588400 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa3438-7d41-42c7-b7df-cd5792016f96-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5\" (UID: \"a8aa3438-7d41-42c7-b7df-cd5792016f96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:45.599197 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.599163 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgwxd\" (UniqueName: \"kubernetes.io/projected/a8aa3438-7d41-42c7-b7df-cd5792016f96-kube-api-access-dgwxd\") pod \"payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5\" (UID: \"a8aa3438-7d41-42c7-b7df-cd5792016f96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:45.599470 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.599452 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krs45\" (UniqueName: \"kubernetes.io/projected/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-kube-api-access-krs45\") pod \"payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz\" (UID: \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:45:45.666300 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.666220 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:45:45.700159 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.700127 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:45.814211 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.814180 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz"] Jul 28 22:45:45.816674 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:45:45.816627 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41869f0d_b3dd_40c8_b124_a6ee3e2015e1.slice/crio-ad99028b12b09552b35bd8b98ad89683a2e2c89469a8921ce9486bcb1af4c0fc WatchSource:0}: Error finding container ad99028b12b09552b35bd8b98ad89683a2e2c89469a8921ce9486bcb1af4c0fc: Status 404 returned error can't find the container with id ad99028b12b09552b35bd8b98ad89683a2e2c89469a8921ce9486bcb1af4c0fc Jul 28 22:45:45.841560 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.841531 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5"] Jul 28 22:45:45.851144 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:45:45.851118 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8aa3438_7d41_42c7_b7df_cd5792016f96.slice/crio-44bf33bf49257d295bbfd9b39acd8193dc35bf972d4c3ab7f6bb3cd7aed00f09 WatchSource:0}: Error finding container 44bf33bf49257d295bbfd9b39acd8193dc35bf972d4c3ab7f6bb3cd7aed00f09: Status 404 returned error can't find the container with id 44bf33bf49257d295bbfd9b39acd8193dc35bf972d4c3ab7f6bb3cd7aed00f09 Jul 28 22:45:45.956462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.956356 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" event={"ID":"41869f0d-b3dd-40c8-b124-a6ee3e2015e1","Type":"ContainerStarted","Data":"d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af"} Jul 28 22:45:45.956462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.956413 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" event={"ID":"41869f0d-b3dd-40c8-b124-a6ee3e2015e1","Type":"ContainerStarted","Data":"ad99028b12b09552b35bd8b98ad89683a2e2c89469a8921ce9486bcb1af4c0fc"} Jul 28 22:45:45.956462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.956441 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:45:45.958104 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.958078 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" event={"ID":"a8aa3438-7d41-42c7-b7df-cd5792016f96","Type":"ContainerStarted","Data":"a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150"} Jul 28 22:45:45.958230 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.958123 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" event={"ID":"a8aa3438-7d41-42c7-b7df-cd5792016f96","Type":"ContainerStarted","Data":"44bf33bf49257d295bbfd9b39acd8193dc35bf972d4c3ab7f6bb3cd7aed00f09"} Jul 28 22:45:45.958230 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.958175 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:45.973176 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.973116 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" podStartSLOduration=0.973095427 podStartE2EDuration="973.095427ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:45:45.971375305 +0000 UTC m=+2470.609234452" watchObservedRunningTime="2026-07-28 22:45:45.973095427 +0000 UTC m=+2470.610954561" Jul 28 22:45:45.989412 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:45.989358 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" podStartSLOduration=0.989344441 podStartE2EDuration="989.344441ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:45:45.987668119 +0000 UTC m=+2470.625527249" watchObservedRunningTime="2026-07-28 22:45:45.989344441 +0000 UTC m=+2470.627203570" Jul 28 22:45:52.195298 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:52.195259 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:45:56.964784 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:56.964753 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:45:56.965178 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:45:56.964834 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:46:20.488327 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:46:20.488292 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:46:40.602457 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:46:40.602424 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:47:06.093578 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:47:06.093542 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:47:26.597447 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:47:26.597417 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:47:44.699473 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:47:44.699436 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:48:12.692328 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:48:12.692251 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:48:29.344453 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:48:29.344418 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:48:51.194531 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:48:51.194493 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:49:04.589098 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:49:04.589065 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:49:16.495208 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:49:16.495167 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:49:34.395671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:49:34.395584 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:49:52.996049 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:49:52.996000 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:50:19.101171 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:50:19.101135 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:50:36.793586 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:50:36.793547 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:51:03.720833 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:51:03.720752 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:51:19.888862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:51:19.888824 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:51:33.194391 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:51:33.194352 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:51:50.201731 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:51:50.201694 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:52:20.893720 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:52:20.893686 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:52:42.391716 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:52:42.391671 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:53:02.096106 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:53:02.096069 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:53:11.491509 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:53:11.491472 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:53:27.298185 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:53:27.298149 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:53:58.300470 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:53:58.300387 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:54:12.690855 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:12.690820 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:54:29.600034 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:29.599995 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:54:44.389028 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:44.388989 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:54:58.462544 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.462503 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz"] Jul 28 22:54:58.466073 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.466055 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:54:58.468956 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.468916 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ff4fa1\"" Jul 28 22:54:58.475302 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.475281 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz"] Jul 28 22:54:58.484339 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.484317 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc"] Jul 28 22:54:58.488352 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.488331 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:54:58.490840 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.490817 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ff4fa1-dockercfg-lbtr2\"" Jul 28 22:54:58.497610 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.497585 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc"] Jul 28 22:54:58.569245 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.569212 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfc9\" (UniqueName: \"kubernetes.io/projected/153e47e8-767c-40f4-998d-485c92de86c6-kube-api-access-znfc9\") pod \"payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz\" (UID: \"153e47e8-767c-40f4-998d-485c92de86c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:54:58.569454 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.569271 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnclt\" (UniqueName: \"kubernetes.io/projected/5b644adf-f27a-4ec4-9cde-10d658501345-kube-api-access-jnclt\") pod \"payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc\" (UID: \"5b644adf-f27a-4ec4-9cde-10d658501345\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:54:58.569454 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.569347 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153e47e8-767c-40f4-998d-485c92de86c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz\" (UID: \"153e47e8-767c-40f4-998d-485c92de86c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:54:58.569454 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.569415 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b644adf-f27a-4ec4-9cde-10d658501345-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc\" (UID: \"5b644adf-f27a-4ec4-9cde-10d658501345\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:54:58.670520 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.670486 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfc9\" (UniqueName: \"kubernetes.io/projected/153e47e8-767c-40f4-998d-485c92de86c6-kube-api-access-znfc9\") pod \"payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz\" (UID: \"153e47e8-767c-40f4-998d-485c92de86c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:54:58.670718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.670532 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnclt\" (UniqueName: \"kubernetes.io/projected/5b644adf-f27a-4ec4-9cde-10d658501345-kube-api-access-jnclt\") pod \"payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc\" (UID: \"5b644adf-f27a-4ec4-9cde-10d658501345\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:54:58.670718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.670552 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153e47e8-767c-40f4-998d-485c92de86c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz\" (UID: \"153e47e8-767c-40f4-998d-485c92de86c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:54:58.670718 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.670685 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b644adf-f27a-4ec4-9cde-10d658501345-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc\" (UID: \"5b644adf-f27a-4ec4-9cde-10d658501345\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:54:58.671499 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.671475 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153e47e8-767c-40f4-998d-485c92de86c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz\" (UID: \"153e47e8-767c-40f4-998d-485c92de86c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:54:58.671576 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.671509 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b644adf-f27a-4ec4-9cde-10d658501345-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc\" (UID: \"5b644adf-f27a-4ec4-9cde-10d658501345\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:54:58.678987 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.678958 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfc9\" (UniqueName: \"kubernetes.io/projected/153e47e8-767c-40f4-998d-485c92de86c6-kube-api-access-znfc9\") pod \"payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz\" (UID: \"153e47e8-767c-40f4-998d-485c92de86c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:54:58.679288 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.679271 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnclt\" (UniqueName: \"kubernetes.io/projected/5b644adf-f27a-4ec4-9cde-10d658501345-kube-api-access-jnclt\") pod \"payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc\" (UID: \"5b644adf-f27a-4ec4-9cde-10d658501345\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:54:58.778068 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.778030 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:54:58.800824 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.800767 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:54:58.926435 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:58.926381 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz"] Jul 28 22:54:58.926837 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:54:58.926794 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153e47e8_767c_40f4_998d_485c92de86c6.slice/crio-eed12729b2d38439044e6933f55756b775e7df0c15bbb9ddfe2720a0f30f8f4f WatchSource:0}: Error finding container eed12729b2d38439044e6933f55756b775e7df0c15bbb9ddfe2720a0f30f8f4f: Status 404 returned error can't find the container with id eed12729b2d38439044e6933f55756b775e7df0c15bbb9ddfe2720a0f30f8f4f Jul 28 22:54:59.162362 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:59.162338 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc"] Jul 28 22:54:59.164413 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:54:59.164390 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b644adf_f27a_4ec4_9cde_10d658501345.slice/crio-302c09f10cb43a47b5586c8d5e70baf80fae05c6354a3ece3e9acd00a4ba6364 WatchSource:0}: Error finding container 302c09f10cb43a47b5586c8d5e70baf80fae05c6354a3ece3e9acd00a4ba6364: Status 404 returned error can't find the container with id 302c09f10cb43a47b5586c8d5e70baf80fae05c6354a3ece3e9acd00a4ba6364 Jul 28 22:54:59.655383 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:59.655319 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" event={"ID":"5b644adf-f27a-4ec4-9cde-10d658501345","Type":"ContainerStarted","Data":"2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828"} Jul 28 22:54:59.655898 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:59.655390 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:54:59.655898 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:59.655419 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" event={"ID":"5b644adf-f27a-4ec4-9cde-10d658501345","Type":"ContainerStarted","Data":"302c09f10cb43a47b5586c8d5e70baf80fae05c6354a3ece3e9acd00a4ba6364"} Jul 28 22:54:59.657090 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:59.657062 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" event={"ID":"153e47e8-767c-40f4-998d-485c92de86c6","Type":"ContainerStarted","Data":"09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe"} Jul 28 22:54:59.657224 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:59.657113 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" event={"ID":"153e47e8-767c-40f4-998d-485c92de86c6","Type":"ContainerStarted","Data":"eed12729b2d38439044e6933f55756b775e7df0c15bbb9ddfe2720a0f30f8f4f"} Jul 28 22:54:59.657224 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:59.657194 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:54:59.672479 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:59.672429 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" podStartSLOduration=1.672412069 podStartE2EDuration="1.672412069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:54:59.670694417 +0000 UTC m=+3024.308553571" watchObservedRunningTime="2026-07-28 22:54:59.672412069 +0000 UTC m=+3024.310271199" Jul 28 22:54:59.685893 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:54:59.685828 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" podStartSLOduration=1.685806438 podStartE2EDuration="1.685806438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:54:59.684891428 +0000 UTC m=+3024.322750557" watchObservedRunningTime="2026-07-28 22:54:59.685806438 +0000 UTC m=+3024.323665567" Jul 28 22:55:06.409977 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:06.409912 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:55:10.664397 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:10.664358 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:55:10.664833 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:10.664669 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:55:27.443038 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.442958 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc"] Jul 28 22:55:27.446888 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.446867 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:27.449244 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.449221 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-056c8a\"" Jul 28 22:55:27.449369 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.449226 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-056c8a-dockercfg-vjdrt\"" Jul 28 22:55:27.453726 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.453692 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc"] Jul 28 22:55:27.480294 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.480252 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t"] Jul 28 22:55:27.483669 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.483646 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:27.494040 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.494020 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t"] Jul 28 22:55:27.545767 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.545738 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkrl\" (UniqueName: \"kubernetes.io/projected/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-kube-api-access-nrkrl\") pod \"payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc\" (UID: \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:27.545961 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.545794 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t\" (UID: \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:27.545961 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.545945 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwp5\" (UniqueName: \"kubernetes.io/projected/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-kube-api-access-4fwp5\") pod \"payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t\" (UID: \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:27.546056 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.545991 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc\" (UID: \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:27.646799 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.646764 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwp5\" (UniqueName: \"kubernetes.io/projected/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-kube-api-access-4fwp5\") pod \"payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t\" (UID: \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:27.646799 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.646803 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc\" (UID: \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:27.647102 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.646863 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkrl\" (UniqueName: \"kubernetes.io/projected/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-kube-api-access-nrkrl\") pod \"payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc\" (UID: \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:27.647102 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.646906 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t\" (UID: \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:27.647620 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.647598 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t\" (UID: \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:27.647620 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.647609 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc\" (UID: \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:27.664082 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.664057 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwp5\" (UniqueName: \"kubernetes.io/projected/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-kube-api-access-4fwp5\") pod \"payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t\" (UID: \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:27.664231 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.664101 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkrl\" (UniqueName: \"kubernetes.io/projected/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-kube-api-access-nrkrl\") pod \"payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc\" (UID: \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:27.759137 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.759101 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:27.795488 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.795463 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:27.909790 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.909739 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc"] Jul 28 22:55:27.911791 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:55:27.911766 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e132bf2_2a7f_4ca0_aaf5_9d5533631c52.slice/crio-4c264b8182ce7fe74c44f34b5805608dd48f82e2c23d47eef6381e065c033bd2 WatchSource:0}: Error finding container 4c264b8182ce7fe74c44f34b5805608dd48f82e2c23d47eef6381e065c033bd2: Status 404 returned error can't find the container with id 4c264b8182ce7fe74c44f34b5805608dd48f82e2c23d47eef6381e065c033bd2 Jul 28 22:55:27.944954 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:27.944915 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t"] Jul 28 22:55:27.950256 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:55:27.950225 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e45d50_4a2d_4643_a6b3_03b7614d1f31.slice/crio-c1cdc5543491b80175c1463eb37486153adfa35decdb0468fe8631abb416ff64 WatchSource:0}: Error finding container c1cdc5543491b80175c1463eb37486153adfa35decdb0468fe8631abb416ff64: Status 404 returned error can't find the container with id c1cdc5543491b80175c1463eb37486153adfa35decdb0468fe8631abb416ff64 Jul 28 22:55:28.804391 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:28.804358 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" event={"ID":"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52","Type":"ContainerStarted","Data":"41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7"} Jul 28 22:55:28.804846 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:28.804440 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:28.804846 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:28.804460 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" event={"ID":"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52","Type":"ContainerStarted","Data":"4c264b8182ce7fe74c44f34b5805608dd48f82e2c23d47eef6381e065c033bd2"} Jul 28 22:55:28.805849 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:28.805827 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" event={"ID":"a6e45d50-4a2d-4643-a6b3-03b7614d1f31","Type":"ContainerStarted","Data":"3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b"} Jul 28 22:55:28.805958 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:28.805865 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" event={"ID":"a6e45d50-4a2d-4643-a6b3-03b7614d1f31","Type":"ContainerStarted","Data":"c1cdc5543491b80175c1463eb37486153adfa35decdb0468fe8631abb416ff64"} Jul 28 22:55:28.805958 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:28.805901 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:28.820334 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:28.820291 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" podStartSLOduration=1.820279161 podStartE2EDuration="1.820279161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:55:28.819223437 +0000 UTC m=+3053.457082567" watchObservedRunningTime="2026-07-28 22:55:28.820279161 +0000 UTC m=+3053.458138289" Jul 28 22:55:28.835109 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:28.835059 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" podStartSLOduration=1.835042895 podStartE2EDuration="1.835042895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:55:28.833828992 +0000 UTC m=+3053.471688124" watchObservedRunningTime="2026-07-28 22:55:28.835042895 +0000 UTC m=+3053.472902024" Jul 28 22:55:36.389030 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:36.388994 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:55:39.813840 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:39.813806 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:55:39.816115 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:39.813966 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:55:58.556455 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:58.556427 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-m7ch8_f207a0bb-d507-481b-934d-04eb809019ec/payload-processing/0.log" Jul 28 22:55:58.685134 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:55:58.685098 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t_a6e45d50-4a2d-4643-a6b3-03b7614d1f31/payload-processing/0.log" Jul 28 22:56:03.895600 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:03.895561 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:56:38.299284 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:38.299245 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:56:48.585167 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:48.585136 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc_5b644adf-f27a-4ec4-9cde-10d658501345/payload-processing/0.log" Jul 28 22:56:48.712977 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:48.712950 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-m7ch8_f207a0bb-d507-481b-934d-04eb809019ec/payload-processing/0.log" Jul 28 22:56:48.833177 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:48.833151 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t_a6e45d50-4a2d-4643-a6b3-03b7614d1f31/payload-processing/0.log" Jul 28 22:56:53.142159 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.142121 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx"] Jul 28 22:56:53.145992 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.145968 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:56:53.149071 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.149048 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5b3501\"" Jul 28 22:56:53.149197 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.149078 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5b3501-dockercfg-h5hrl\"" Jul 28 22:56:53.156160 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.156138 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx"] Jul 28 22:56:53.167318 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.167279 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v"] Jul 28 22:56:53.171030 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.171003 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:56:53.183160 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.183132 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v"] Jul 28 22:56:53.286406 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.286369 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f3e8fc-b06e-4d86-8f01-75c216938337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v\" (UID: \"03f3e8fc-b06e-4d86-8f01-75c216938337\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:56:53.286612 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.286429 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3ee3fa-6013-42f2-986e-6b761badf0b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx\" (UID: \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:56:53.286612 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.286506 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4rkf\" (UniqueName: \"kubernetes.io/projected/03f3e8fc-b06e-4d86-8f01-75c216938337-kube-api-access-d4rkf\") pod \"payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v\" (UID: \"03f3e8fc-b06e-4d86-8f01-75c216938337\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:56:53.286612 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.286594 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp79x\" (UniqueName: \"kubernetes.io/projected/2e3ee3fa-6013-42f2-986e-6b761badf0b3-kube-api-access-tp79x\") pod \"payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx\" (UID: \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:56:53.387879 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.387831 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp79x\" (UniqueName: \"kubernetes.io/projected/2e3ee3fa-6013-42f2-986e-6b761badf0b3-kube-api-access-tp79x\") pod \"payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx\" (UID: \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:56:53.388124 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.387966 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f3e8fc-b06e-4d86-8f01-75c216938337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v\" (UID: \"03f3e8fc-b06e-4d86-8f01-75c216938337\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:56:53.388124 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.388005 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3ee3fa-6013-42f2-986e-6b761badf0b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx\" (UID: \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:56:53.388124 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.388030 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4rkf\" (UniqueName: \"kubernetes.io/projected/03f3e8fc-b06e-4d86-8f01-75c216938337-kube-api-access-d4rkf\") pod \"payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v\" (UID: \"03f3e8fc-b06e-4d86-8f01-75c216938337\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:56:53.388701 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.388681 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3ee3fa-6013-42f2-986e-6b761badf0b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx\" (UID: \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:56:53.388750 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.388687 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f3e8fc-b06e-4d86-8f01-75c216938337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v\" (UID: \"03f3e8fc-b06e-4d86-8f01-75c216938337\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:56:53.398301 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.398235 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4rkf\" (UniqueName: \"kubernetes.io/projected/03f3e8fc-b06e-4d86-8f01-75c216938337-kube-api-access-d4rkf\") pod \"payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v\" (UID: \"03f3e8fc-b06e-4d86-8f01-75c216938337\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:56:53.398413 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.398366 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp79x\" (UniqueName: \"kubernetes.io/projected/2e3ee3fa-6013-42f2-986e-6b761badf0b3-kube-api-access-tp79x\") pod \"payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx\" (UID: \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:56:53.458329 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.458295 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:56:53.485509 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.485479 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:56:53.609297 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.609271 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx"] Jul 28 22:56:53.611164 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:56:53.611134 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3ee3fa_6013_42f2_986e_6b761badf0b3.slice/crio-afd145eb56539536b00afc5cde9571061ef8b21fa08fc126a449b13879c52604 WatchSource:0}: Error finding container afd145eb56539536b00afc5cde9571061ef8b21fa08fc126a449b13879c52604: Status 404 returned error can't find the container with id afd145eb56539536b00afc5cde9571061ef8b21fa08fc126a449b13879c52604 Jul 28 22:56:53.848746 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:53.848722 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v"] Jul 28 22:56:53.850784 ip-10-0-142-225 kubenswrapper[2557]: W0728 22:56:53.850753 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f3e8fc_b06e_4d86_8f01_75c216938337.slice/crio-6fa48ae30597f03b7394d777081cb3adae2fe465db3ffe37808950e1b5abded7 WatchSource:0}: Error finding container 6fa48ae30597f03b7394d777081cb3adae2fe465db3ffe37808950e1b5abded7: Status 404 returned error can't find the container with id 6fa48ae30597f03b7394d777081cb3adae2fe465db3ffe37808950e1b5abded7 Jul 28 22:56:54.243196 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:54.243157 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" event={"ID":"2e3ee3fa-6013-42f2-986e-6b761badf0b3","Type":"ContainerStarted","Data":"645d8c2bc83480e8e6285d3249fe10bfde9c1c563f630efa6c68b791df8391bb"} Jul 28 22:56:54.243670 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:54.243222 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" event={"ID":"2e3ee3fa-6013-42f2-986e-6b761badf0b3","Type":"ContainerStarted","Data":"afd145eb56539536b00afc5cde9571061ef8b21fa08fc126a449b13879c52604"} Jul 28 22:56:54.243670 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:54.243252 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:56:54.244963 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:54.244914 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" event={"ID":"03f3e8fc-b06e-4d86-8f01-75c216938337","Type":"ContainerStarted","Data":"8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28"} Jul 28 22:56:54.245107 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:54.244979 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" event={"ID":"03f3e8fc-b06e-4d86-8f01-75c216938337","Type":"ContainerStarted","Data":"6fa48ae30597f03b7394d777081cb3adae2fe465db3ffe37808950e1b5abded7"} Jul 28 22:56:54.245107 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:54.245024 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:56:54.262408 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:54.262346 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" podStartSLOduration=1.262324384 podStartE2EDuration="1.262324384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:56:54.260414163 +0000 UTC m=+3138.898273314" watchObservedRunningTime="2026-07-28 22:56:54.262324384 +0000 UTC m=+3138.900183494" Jul 28 22:56:54.277357 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:54.277295 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" podStartSLOduration=1.277274837 podStartE2EDuration="1.277274837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:56:54.276807186 +0000 UTC m=+3138.914666316" watchObservedRunningTime="2026-07-28 22:56:54.277274837 +0000 UTC m=+3138.915133972" Jul 28 22:56:57.587448 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:56:57.587356 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:57:05.252656 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:05.252618 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:57:05.253050 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:05.252690 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:57:23.084854 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.084814 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v"] Jul 28 22:57:23.085739 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.085697 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" podUID="03f3e8fc-b06e-4d86-8f01-75c216938337" containerName="payload-processing" containerID="cri-o://8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28" gracePeriod=30 Jul 28 22:57:23.087671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.087420 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx"] Jul 28 22:57:23.087671 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.087615 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" podUID="2e3ee3fa-6013-42f2-986e-6b761badf0b3" containerName="payload-pre-processing" containerID="cri-o://645d8c2bc83480e8e6285d3249fe10bfde9c1c563f630efa6c68b791df8391bb" gracePeriod=30 Jul 28 22:57:23.380556 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.380525 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:57:23.397253 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.397218 2557 generic.go:356] "Generic (PLEG): container finished" podID="03f3e8fc-b06e-4d86-8f01-75c216938337" containerID="8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28" exitCode=0 Jul 28 22:57:23.397426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.397270 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" Jul 28 22:57:23.397426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.397298 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" event={"ID":"03f3e8fc-b06e-4d86-8f01-75c216938337","Type":"ContainerDied","Data":"8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28"} Jul 28 22:57:23.397426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.397363 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v" event={"ID":"03f3e8fc-b06e-4d86-8f01-75c216938337","Type":"ContainerDied","Data":"6fa48ae30597f03b7394d777081cb3adae2fe465db3ffe37808950e1b5abded7"} Jul 28 22:57:23.397426 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.397394 2557 scope.go:117] "RemoveContainer" containerID="8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28" Jul 28 22:57:23.400456 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.400326 2557 generic.go:356] "Generic (PLEG): container finished" podID="2e3ee3fa-6013-42f2-986e-6b761badf0b3" containerID="645d8c2bc83480e8e6285d3249fe10bfde9c1c563f630efa6c68b791df8391bb" exitCode=0 Jul 28 22:57:23.400456 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.400369 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" event={"ID":"2e3ee3fa-6013-42f2-986e-6b761badf0b3","Type":"ContainerDied","Data":"645d8c2bc83480e8e6285d3249fe10bfde9c1c563f630efa6c68b791df8391bb"} Jul 28 22:57:23.400456 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.400427 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" event={"ID":"2e3ee3fa-6013-42f2-986e-6b761badf0b3","Type":"ContainerDied","Data":"afd145eb56539536b00afc5cde9571061ef8b21fa08fc126a449b13879c52604"} Jul 28 22:57:23.400456 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.400445 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd145eb56539536b00afc5cde9571061ef8b21fa08fc126a449b13879c52604" Jul 28 22:57:23.402491 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.402461 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:57:23.410531 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.410504 2557 scope.go:117] "RemoveContainer" containerID="8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28" Jul 28 22:57:23.410826 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:57:23.410808 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28\": container with ID starting with 8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28 not found: ID does not exist" containerID="8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28" Jul 28 22:57:23.410908 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.410835 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28"} err="failed to get container status \"8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28\": rpc error: code = NotFound desc = could not find container \"8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28\": container with ID starting with 8add54d3fee51ba6b313685bc0eeddc18ac23c3b87729cdd18fe8f93272f2e28 not found: ID does not exist" Jul 28 22:57:23.472787 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.472755 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4rkf\" (UniqueName: \"kubernetes.io/projected/03f3e8fc-b06e-4d86-8f01-75c216938337-kube-api-access-d4rkf\") pod \"03f3e8fc-b06e-4d86-8f01-75c216938337\" (UID: \"03f3e8fc-b06e-4d86-8f01-75c216938337\") " Jul 28 22:57:23.473007 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.472807 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp79x\" (UniqueName: \"kubernetes.io/projected/2e3ee3fa-6013-42f2-986e-6b761badf0b3-kube-api-access-tp79x\") pod \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\" (UID: \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\") " Jul 28 22:57:23.473007 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.472863 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3ee3fa-6013-42f2-986e-6b761badf0b3-plugins-config-volume\") pod \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\" (UID: \"2e3ee3fa-6013-42f2-986e-6b761badf0b3\") " Jul 28 22:57:23.473007 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.472907 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f3e8fc-b06e-4d86-8f01-75c216938337-plugins-config-volume\") pod \"03f3e8fc-b06e-4d86-8f01-75c216938337\" (UID: \"03f3e8fc-b06e-4d86-8f01-75c216938337\") " Jul 28 22:57:23.473293 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.473269 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3ee3fa-6013-42f2-986e-6b761badf0b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e3ee3fa-6013-42f2-986e-6b761badf0b3" (UID: "2e3ee3fa-6013-42f2-986e-6b761badf0b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:57:23.473356 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.473290 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f3e8fc-b06e-4d86-8f01-75c216938337-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03f3e8fc-b06e-4d86-8f01-75c216938337" (UID: "03f3e8fc-b06e-4d86-8f01-75c216938337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:57:23.475123 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.475092 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3ee3fa-6013-42f2-986e-6b761badf0b3-kube-api-access-tp79x" (OuterVolumeSpecName: "kube-api-access-tp79x") pod "2e3ee3fa-6013-42f2-986e-6b761badf0b3" (UID: "2e3ee3fa-6013-42f2-986e-6b761badf0b3"). InnerVolumeSpecName "kube-api-access-tp79x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:57:23.475222 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.475133 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f3e8fc-b06e-4d86-8f01-75c216938337-kube-api-access-d4rkf" (OuterVolumeSpecName: "kube-api-access-d4rkf") pod "03f3e8fc-b06e-4d86-8f01-75c216938337" (UID: "03f3e8fc-b06e-4d86-8f01-75c216938337"). InnerVolumeSpecName "kube-api-access-d4rkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:57:23.574624 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.574586 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4rkf\" (UniqueName: \"kubernetes.io/projected/03f3e8fc-b06e-4d86-8f01-75c216938337-kube-api-access-d4rkf\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:57:23.574624 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.574618 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp79x\" (UniqueName: \"kubernetes.io/projected/2e3ee3fa-6013-42f2-986e-6b761badf0b3-kube-api-access-tp79x\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:57:23.574840 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.574632 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3ee3fa-6013-42f2-986e-6b761badf0b3-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:57:23.574840 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.574644 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03f3e8fc-b06e-4d86-8f01-75c216938337-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:57:23.727697 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.727666 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v"] Jul 28 22:57:23.735020 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:23.734989 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5b3501-66bc5698b4-vvw9v"] Jul 28 22:57:24.046981 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:24.046920 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f3e8fc-b06e-4d86-8f01-75c216938337" path="/var/lib/kubelet/pods/03f3e8fc-b06e-4d86-8f01-75c216938337/volumes" Jul 28 22:57:24.406106 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:24.406022 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx" Jul 28 22:57:24.422817 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:24.422778 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx"] Jul 28 22:57:24.426878 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:24.426851 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5b3501-65c864c84-mjfjx"] Jul 28 22:57:26.046027 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:26.045992 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e3ee3fa-6013-42f2-986e-6b761badf0b3" path="/var/lib/kubelet/pods/2e3ee3fa-6013-42f2-986e-6b761badf0b3/volumes" Jul 28 22:57:27.792624 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:27.792590 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:57:47.596081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:47.596042 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:57:48.886520 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:48.886485 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc"] Jul 28 22:57:48.887032 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:48.886702 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" podUID="5b644adf-f27a-4ec4-9cde-10d658501345" containerName="payload-processing" containerID="cri-o://2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828" gracePeriod=30 Jul 28 22:57:48.893151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:48.893119 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz"] Jul 28 22:57:48.893425 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:48.893382 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" podUID="153e47e8-767c-40f4-998d-485c92de86c6" containerName="payload-pre-processing" containerID="cri-o://09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe" gracePeriod=30 Jul 28 22:57:49.183833 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.183804 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:57:49.187036 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.187011 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:57:49.230151 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.230121 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153e47e8-767c-40f4-998d-485c92de86c6-plugins-config-volume\") pod \"153e47e8-767c-40f4-998d-485c92de86c6\" (UID: \"153e47e8-767c-40f4-998d-485c92de86c6\") " Jul 28 22:57:49.230350 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.230175 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnclt\" (UniqueName: \"kubernetes.io/projected/5b644adf-f27a-4ec4-9cde-10d658501345-kube-api-access-jnclt\") pod \"5b644adf-f27a-4ec4-9cde-10d658501345\" (UID: \"5b644adf-f27a-4ec4-9cde-10d658501345\") " Jul 28 22:57:49.230350 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.230222 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b644adf-f27a-4ec4-9cde-10d658501345-plugins-config-volume\") pod \"5b644adf-f27a-4ec4-9cde-10d658501345\" (UID: \"5b644adf-f27a-4ec4-9cde-10d658501345\") " Jul 28 22:57:49.230350 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.230250 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znfc9\" (UniqueName: \"kubernetes.io/projected/153e47e8-767c-40f4-998d-485c92de86c6-kube-api-access-znfc9\") pod \"153e47e8-767c-40f4-998d-485c92de86c6\" (UID: \"153e47e8-767c-40f4-998d-485c92de86c6\") " Jul 28 22:57:49.230642 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.230615 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b644adf-f27a-4ec4-9cde-10d658501345-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b644adf-f27a-4ec4-9cde-10d658501345" (UID: "5b644adf-f27a-4ec4-9cde-10d658501345"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:57:49.230642 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.230626 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153e47e8-767c-40f4-998d-485c92de86c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "153e47e8-767c-40f4-998d-485c92de86c6" (UID: "153e47e8-767c-40f4-998d-485c92de86c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:57:49.232392 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.232364 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b644adf-f27a-4ec4-9cde-10d658501345-kube-api-access-jnclt" (OuterVolumeSpecName: "kube-api-access-jnclt") pod "5b644adf-f27a-4ec4-9cde-10d658501345" (UID: "5b644adf-f27a-4ec4-9cde-10d658501345"). InnerVolumeSpecName "kube-api-access-jnclt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:57:49.232525 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.232402 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153e47e8-767c-40f4-998d-485c92de86c6-kube-api-access-znfc9" (OuterVolumeSpecName: "kube-api-access-znfc9") pod "153e47e8-767c-40f4-998d-485c92de86c6" (UID: "153e47e8-767c-40f4-998d-485c92de86c6"). InnerVolumeSpecName "kube-api-access-znfc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:57:49.331081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.331038 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b644adf-f27a-4ec4-9cde-10d658501345-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:57:49.331081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.331073 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znfc9\" (UniqueName: \"kubernetes.io/projected/153e47e8-767c-40f4-998d-485c92de86c6-kube-api-access-znfc9\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:57:49.331081 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.331088 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153e47e8-767c-40f4-998d-485c92de86c6-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:57:49.331345 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.331102 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnclt\" (UniqueName: \"kubernetes.io/projected/5b644adf-f27a-4ec4-9cde-10d658501345-kube-api-access-jnclt\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:57:49.539143 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.539110 2557 generic.go:356] "Generic (PLEG): container finished" podID="5b644adf-f27a-4ec4-9cde-10d658501345" containerID="2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828" exitCode=0 Jul 28 22:57:49.539361 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.539171 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" Jul 28 22:57:49.539361 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.539203 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" event={"ID":"5b644adf-f27a-4ec4-9cde-10d658501345","Type":"ContainerDied","Data":"2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828"} Jul 28 22:57:49.539361 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.539258 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc" event={"ID":"5b644adf-f27a-4ec4-9cde-10d658501345","Type":"ContainerDied","Data":"302c09f10cb43a47b5586c8d5e70baf80fae05c6354a3ece3e9acd00a4ba6364"} Jul 28 22:57:49.539361 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.539289 2557 scope.go:117] "RemoveContainer" containerID="2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828" Jul 28 22:57:49.540648 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.540627 2557 generic.go:356] "Generic (PLEG): container finished" podID="153e47e8-767c-40f4-998d-485c92de86c6" containerID="09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe" exitCode=0 Jul 28 22:57:49.540736 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.540667 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" Jul 28 22:57:49.540736 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.540694 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" event={"ID":"153e47e8-767c-40f4-998d-485c92de86c6","Type":"ContainerDied","Data":"09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe"} Jul 28 22:57:49.540837 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.540736 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz" event={"ID":"153e47e8-767c-40f4-998d-485c92de86c6","Type":"ContainerDied","Data":"eed12729b2d38439044e6933f55756b775e7df0c15bbb9ddfe2720a0f30f8f4f"} Jul 28 22:57:49.549203 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.549178 2557 scope.go:117] "RemoveContainer" containerID="2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828" Jul 28 22:57:49.549529 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:57:49.549513 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828\": container with ID starting with 2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828 not found: ID does not exist" containerID="2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828" Jul 28 22:57:49.549591 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.549539 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828"} err="failed to get container status \"2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828\": rpc error: code = NotFound desc = could not find container \"2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828\": container with ID starting with 2416506e5fda0fc8ad202898e1cbb51e106019283cf059ca2c09a5e089b66828 not found: ID does not exist" Jul 28 22:57:49.549591 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.549554 2557 scope.go:117] "RemoveContainer" containerID="09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe" Jul 28 22:57:49.559476 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.559448 2557 scope.go:117] "RemoveContainer" containerID="09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe" Jul 28 22:57:49.559800 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:57:49.559777 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe\": container with ID starting with 09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe not found: ID does not exist" containerID="09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe" Jul 28 22:57:49.559845 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.559810 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe"} err="failed to get container status \"09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe\": rpc error: code = NotFound desc = could not find container \"09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe\": container with ID starting with 09577f2fc69d2489dfccc41911f155ef428fae08e38f34b5204787bd2bdc80fe not found: ID does not exist" Jul 28 22:57:49.563555 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.563529 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz"] Jul 28 22:57:49.569422 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.569396 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff4fa1-5d4dc4974d-6r5cz"] Jul 28 22:57:49.580254 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.580225 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc"] Jul 28 22:57:49.585532 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:49.585503 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff4fa1-664f7b958b-zlcdc"] Jul 28 22:57:50.046568 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:50.046534 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="153e47e8-767c-40f4-998d-485c92de86c6" path="/var/lib/kubelet/pods/153e47e8-767c-40f4-998d-485c92de86c6/volumes" Jul 28 22:57:50.046964 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:50.046901 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b644adf-f27a-4ec4-9cde-10d658501345" path="/var/lib/kubelet/pods/5b644adf-f27a-4ec4-9cde-10d658501345/volumes" Jul 28 22:57:58.892053 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:57:58.892016 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:58:49.503016 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.502974 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t"] Jul 28 22:58:49.503647 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.503203 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" podUID="a6e45d50-4a2d-4643-a6b3-03b7614d1f31" containerName="payload-processing" containerID="cri-o://3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b" gracePeriod=30 Jul 28 22:58:49.513411 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.513371 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc"] Jul 28 22:58:49.513827 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.513787 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" podUID="8e132bf2-2a7f-4ca0-aaf5-9d5533631c52" containerName="payload-pre-processing" containerID="cri-o://41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7" gracePeriod=30 Jul 28 22:58:49.820464 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.820440 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:58:49.823419 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.823395 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:58:49.845510 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.845480 2557 generic.go:356] "Generic (PLEG): container finished" podID="8e132bf2-2a7f-4ca0-aaf5-9d5533631c52" containerID="41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7" exitCode=0 Jul 28 22:58:49.845682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.845522 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" Jul 28 22:58:49.845682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.845559 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" event={"ID":"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52","Type":"ContainerDied","Data":"41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7"} Jul 28 22:58:49.845682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.845609 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" event={"ID":"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52","Type":"ContainerDied","Data":"4c264b8182ce7fe74c44f34b5805608dd48f82e2c23d47eef6381e065c033bd2"} Jul 28 22:58:49.845682 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.845638 2557 scope.go:117] "RemoveContainer" containerID="41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7" Jul 28 22:58:49.846898 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.846875 2557 generic.go:356] "Generic (PLEG): container finished" podID="a6e45d50-4a2d-4643-a6b3-03b7614d1f31" containerID="3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b" exitCode=0 Jul 28 22:58:49.847023 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.846954 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" Jul 28 22:58:49.847086 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.847036 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" event={"ID":"a6e45d50-4a2d-4643-a6b3-03b7614d1f31","Type":"ContainerDied","Data":"3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b"} Jul 28 22:58:49.847086 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.847080 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" event={"ID":"a6e45d50-4a2d-4643-a6b3-03b7614d1f31","Type":"ContainerDied","Data":"c1cdc5543491b80175c1463eb37486153adfa35decdb0468fe8631abb416ff64"} Jul 28 22:58:49.856248 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.856188 2557 scope.go:117] "RemoveContainer" containerID="41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7" Jul 28 22:58:49.856688 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:58:49.856661 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7\": container with ID starting with 41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7 not found: ID does not exist" containerID="41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7" Jul 28 22:58:49.856790 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.856698 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7"} err="failed to get container status \"41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7\": rpc error: code = NotFound desc = could not find container \"41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7\": container with ID starting with 41cd4e2ce7c0b58500622305d397d0de7e355e6cf311699ee32857e4c9fbd5e7 not found: ID does not exist" Jul 28 22:58:49.856790 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.856721 2557 scope.go:117] "RemoveContainer" containerID="3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b" Jul 28 22:58:49.865741 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.865720 2557 scope.go:117] "RemoveContainer" containerID="3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b" Jul 28 22:58:49.866064 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:58:49.866046 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b\": container with ID starting with 3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b not found: ID does not exist" containerID="3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b" Jul 28 22:58:49.866132 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.866071 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b"} err="failed to get container status \"3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b\": rpc error: code = NotFound desc = could not find container \"3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b\": container with ID starting with 3a80c8718bf82ebd2779949002e06cf8bea948fe0e2f76a8e1c4e7285eb1da6b not found: ID does not exist" Jul 28 22:58:49.911224 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.911189 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fwp5\" (UniqueName: \"kubernetes.io/projected/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-kube-api-access-4fwp5\") pod \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\" (UID: \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\") " Jul 28 22:58:49.911410 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.911269 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-plugins-config-volume\") pod \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\" (UID: \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\") " Jul 28 22:58:49.911484 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.911437 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrkrl\" (UniqueName: \"kubernetes.io/projected/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-kube-api-access-nrkrl\") pod \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\" (UID: \"8e132bf2-2a7f-4ca0-aaf5-9d5533631c52\") " Jul 28 22:58:49.911539 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.911488 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-plugins-config-volume\") pod \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\" (UID: \"a6e45d50-4a2d-4643-a6b3-03b7614d1f31\") " Jul 28 22:58:49.911594 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.911552 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e132bf2-2a7f-4ca0-aaf5-9d5533631c52" (UID: "8e132bf2-2a7f-4ca0-aaf5-9d5533631c52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:58:49.911836 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.911818 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:58:49.911908 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.911872 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6e45d50-4a2d-4643-a6b3-03b7614d1f31" (UID: "a6e45d50-4a2d-4643-a6b3-03b7614d1f31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:58:49.913420 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.913398 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-kube-api-access-4fwp5" (OuterVolumeSpecName: "kube-api-access-4fwp5") pod "a6e45d50-4a2d-4643-a6b3-03b7614d1f31" (UID: "a6e45d50-4a2d-4643-a6b3-03b7614d1f31"). InnerVolumeSpecName "kube-api-access-4fwp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:58:49.913487 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:49.913424 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-kube-api-access-nrkrl" (OuterVolumeSpecName: "kube-api-access-nrkrl") pod "8e132bf2-2a7f-4ca0-aaf5-9d5533631c52" (UID: "8e132bf2-2a7f-4ca0-aaf5-9d5533631c52"). InnerVolumeSpecName "kube-api-access-nrkrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:58:50.012862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:50.012818 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwp5\" (UniqueName: \"kubernetes.io/projected/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-kube-api-access-4fwp5\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:58:50.012862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:50.012852 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrkrl\" (UniqueName: \"kubernetes.io/projected/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52-kube-api-access-nrkrl\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:58:50.012862 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:50.012862 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e45d50-4a2d-4643-a6b3-03b7614d1f31-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:58:50.164141 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:50.164106 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc"] Jul 28 22:58:50.167265 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:50.167238 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc"] Jul 28 22:58:50.178243 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:50.178218 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t"] Jul 28 22:58:50.179881 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:50.179859 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t"] Jul 28 22:58:50.812102 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:50.812056 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-056c8a-7465bd5cfb-g6bmc" podUID="8e132bf2-2a7f-4ca0-aaf5-9d5533631c52" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.58:9004: i/o timeout" Jul 28 22:58:50.812514 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:50.812064 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-056c8a-5cd577f58-9rq2t" podUID="a6e45d50-4a2d-4643-a6b3-03b7614d1f31" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.59:9004: i/o timeout" Jul 28 22:58:52.045938 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:52.045899 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e132bf2-2a7f-4ca0-aaf5-9d5533631c52" path="/var/lib/kubelet/pods/8e132bf2-2a7f-4ca0-aaf5-9d5533631c52/volumes" Jul 28 22:58:52.046338 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:52.046276 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6e45d50-4a2d-4643-a6b3-03b7614d1f31" path="/var/lib/kubelet/pods/a6e45d50-4a2d-4643-a6b3-03b7614d1f31/volumes" Jul 28 22:58:52.497706 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:58:52.497677 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:59:07.076625 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.076588 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m"] Jul 28 22:59:07.077132 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.076795 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" podUID="ae5d913f-599e-4f98-9dc9-4b63a036a851" containerName="payload-processing" containerID="cri-o://0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114" gracePeriod=30 Jul 28 22:59:07.089559 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.089523 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m"] Jul 28 22:59:07.090077 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.089909 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" podUID="f18d3719-7633-4a89-ae74-d59abb0177c1" containerName="payload-pre-processing" containerID="cri-o://a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6" gracePeriod=30 Jul 28 22:59:07.371849 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.371815 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:59:07.374606 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.374587 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:59:07.489215 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.489177 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18d3719-7633-4a89-ae74-d59abb0177c1-plugins-config-volume\") pod \"f18d3719-7633-4a89-ae74-d59abb0177c1\" (UID: \"f18d3719-7633-4a89-ae74-d59abb0177c1\") " Jul 28 22:59:07.489421 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.489254 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5gdc\" (UniqueName: \"kubernetes.io/projected/f18d3719-7633-4a89-ae74-d59abb0177c1-kube-api-access-z5gdc\") pod \"f18d3719-7633-4a89-ae74-d59abb0177c1\" (UID: \"f18d3719-7633-4a89-ae74-d59abb0177c1\") " Jul 28 22:59:07.489421 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.489352 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bvb8\" (UniqueName: \"kubernetes.io/projected/ae5d913f-599e-4f98-9dc9-4b63a036a851-kube-api-access-2bvb8\") pod \"ae5d913f-599e-4f98-9dc9-4b63a036a851\" (UID: \"ae5d913f-599e-4f98-9dc9-4b63a036a851\") " Jul 28 22:59:07.489538 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.489435 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5d913f-599e-4f98-9dc9-4b63a036a851-plugins-config-volume\") pod \"ae5d913f-599e-4f98-9dc9-4b63a036a851\" (UID: \"ae5d913f-599e-4f98-9dc9-4b63a036a851\") " Jul 28 22:59:07.489620 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.489594 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18d3719-7633-4a89-ae74-d59abb0177c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f18d3719-7633-4a89-ae74-d59abb0177c1" (UID: "f18d3719-7633-4a89-ae74-d59abb0177c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:59:07.489805 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.489782 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5d913f-599e-4f98-9dc9-4b63a036a851-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae5d913f-599e-4f98-9dc9-4b63a036a851" (UID: "ae5d913f-599e-4f98-9dc9-4b63a036a851"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:59:07.489884 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.489795 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18d3719-7633-4a89-ae74-d59abb0177c1-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:59:07.491379 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.491354 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18d3719-7633-4a89-ae74-d59abb0177c1-kube-api-access-z5gdc" (OuterVolumeSpecName: "kube-api-access-z5gdc") pod "f18d3719-7633-4a89-ae74-d59abb0177c1" (UID: "f18d3719-7633-4a89-ae74-d59abb0177c1"). InnerVolumeSpecName "kube-api-access-z5gdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:59:07.491379 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.491374 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5d913f-599e-4f98-9dc9-4b63a036a851-kube-api-access-2bvb8" (OuterVolumeSpecName: "kube-api-access-2bvb8") pod "ae5d913f-599e-4f98-9dc9-4b63a036a851" (UID: "ae5d913f-599e-4f98-9dc9-4b63a036a851"). InnerVolumeSpecName "kube-api-access-2bvb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:59:07.590619 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.590581 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bvb8\" (UniqueName: \"kubernetes.io/projected/ae5d913f-599e-4f98-9dc9-4b63a036a851-kube-api-access-2bvb8\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:59:07.590619 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.590614 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5d913f-599e-4f98-9dc9-4b63a036a851-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:59:07.590619 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.590623 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5gdc\" (UniqueName: \"kubernetes.io/projected/f18d3719-7633-4a89-ae74-d59abb0177c1-kube-api-access-z5gdc\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:59:07.942673 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.942640 2557 generic.go:356] "Generic (PLEG): container finished" podID="ae5d913f-599e-4f98-9dc9-4b63a036a851" containerID="0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114" exitCode=0 Jul 28 22:59:07.942858 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.942683 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" Jul 28 22:59:07.942858 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.942728 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" event={"ID":"ae5d913f-599e-4f98-9dc9-4b63a036a851","Type":"ContainerDied","Data":"0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114"} Jul 28 22:59:07.942858 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.942776 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m" event={"ID":"ae5d913f-599e-4f98-9dc9-4b63a036a851","Type":"ContainerDied","Data":"55b00ce0ed0caa1f41e464488dd27841201616edef6855bf5e3fb4feb69df163"} Jul 28 22:59:07.942858 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.942798 2557 scope.go:117] "RemoveContainer" containerID="0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114" Jul 28 22:59:07.944043 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.944022 2557 generic.go:356] "Generic (PLEG): container finished" podID="f18d3719-7633-4a89-ae74-d59abb0177c1" containerID="a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6" exitCode=0 Jul 28 22:59:07.944133 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.944060 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" Jul 28 22:59:07.944133 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.944084 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" event={"ID":"f18d3719-7633-4a89-ae74-d59abb0177c1","Type":"ContainerDied","Data":"a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6"} Jul 28 22:59:07.944133 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.944126 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m" event={"ID":"f18d3719-7633-4a89-ae74-d59abb0177c1","Type":"ContainerDied","Data":"b1c2eb4f7cb84ebfd02354bf9977a16d67666299ae1d3384cb154826711b680f"} Jul 28 22:59:07.952024 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.951975 2557 scope.go:117] "RemoveContainer" containerID="0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114" Jul 28 22:59:07.952286 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:59:07.952261 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114\": container with ID starting with 0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114 not found: ID does not exist" containerID="0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114" Jul 28 22:59:07.952352 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.952290 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114"} err="failed to get container status \"0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114\": rpc error: code = NotFound desc = could not find container \"0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114\": container with ID starting with 0692a0aabb751c3dbd5d33a245d46eef6650d7025ea23d803063798993db0114 not found: ID does not exist" Jul 28 22:59:07.952352 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.952312 2557 scope.go:117] "RemoveContainer" containerID="a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6" Jul 28 22:59:07.960148 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.960131 2557 scope.go:117] "RemoveContainer" containerID="a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6" Jul 28 22:59:07.960402 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:59:07.960382 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6\": container with ID starting with a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6 not found: ID does not exist" containerID="a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6" Jul 28 22:59:07.960454 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.960409 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6"} err="failed to get container status \"a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6\": rpc error: code = NotFound desc = could not find container \"a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6\": container with ID starting with a1c46be5f6ea61f634b888d3b1544e246619eeb8a032d603e775fe74e348a2e6 not found: ID does not exist" Jul 28 22:59:07.966913 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.966890 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m"] Jul 28 22:59:07.972446 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.972414 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ae4fa9-785f49ff5d-c2g8m"] Jul 28 22:59:07.983497 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.983475 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m"] Jul 28 22:59:07.989236 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:07.989212 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ae4fa9-f66f99f5b-km24m"] Jul 28 22:59:08.046637 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:08.046607 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae5d913f-599e-4f98-9dc9-4b63a036a851" path="/var/lib/kubelet/pods/ae5d913f-599e-4f98-9dc9-4b63a036a851/volumes" Jul 28 22:59:08.046999 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:08.046985 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f18d3719-7633-4a89-ae74-d59abb0177c1" path="/var/lib/kubelet/pods/f18d3719-7633-4a89-ae74-d59abb0177c1/volumes" Jul 28 22:59:10.092055 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:10.092020 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:59:24.722624 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:24.722589 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5"] Jul 28 22:59:24.723207 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:24.722840 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" podUID="a8aa3438-7d41-42c7-b7df-cd5792016f96" containerName="payload-processing" containerID="cri-o://a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150" gracePeriod=30 Jul 28 22:59:24.729824 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:24.729796 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz"] Jul 28 22:59:24.730060 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:24.730014 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" podUID="41869f0d-b3dd-40c8-b124-a6ee3e2015e1" containerName="payload-pre-processing" containerID="cri-o://d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af" gracePeriod=30 Jul 28 22:59:24.978514 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:24.978445 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:59:25.010282 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.010253 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:59:25.042831 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.042800 2557 generic.go:356] "Generic (PLEG): container finished" podID="a8aa3438-7d41-42c7-b7df-cd5792016f96" containerID="a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150" exitCode=0 Jul 28 22:59:25.043045 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.042857 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" Jul 28 22:59:25.043045 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.042899 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" event={"ID":"a8aa3438-7d41-42c7-b7df-cd5792016f96","Type":"ContainerDied","Data":"a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150"} Jul 28 22:59:25.043045 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.042964 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5" event={"ID":"a8aa3438-7d41-42c7-b7df-cd5792016f96","Type":"ContainerDied","Data":"44bf33bf49257d295bbfd9b39acd8193dc35bf972d4c3ab7f6bb3cd7aed00f09"} Jul 28 22:59:25.043045 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.042996 2557 scope.go:117] "RemoveContainer" containerID="a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150" Jul 28 22:59:25.044295 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.044270 2557 generic.go:356] "Generic (PLEG): container finished" podID="41869f0d-b3dd-40c8-b124-a6ee3e2015e1" containerID="d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af" exitCode=0 Jul 28 22:59:25.044392 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.044316 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" Jul 28 22:59:25.044454 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.044316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" event={"ID":"41869f0d-b3dd-40c8-b124-a6ee3e2015e1","Type":"ContainerDied","Data":"d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af"} Jul 28 22:59:25.044454 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.044445 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz" event={"ID":"41869f0d-b3dd-40c8-b124-a6ee3e2015e1","Type":"ContainerDied","Data":"ad99028b12b09552b35bd8b98ad89683a2e2c89469a8921ce9486bcb1af4c0fc"} Jul 28 22:59:25.051402 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.051383 2557 scope.go:117] "RemoveContainer" containerID="a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150" Jul 28 22:59:25.051675 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:59:25.051657 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150\": container with ID starting with a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150 not found: ID does not exist" containerID="a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150" Jul 28 22:59:25.051731 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.051685 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150"} err="failed to get container status \"a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150\": rpc error: code = NotFound desc = could not find container \"a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150\": container with ID starting with a28a8072f14c694da70bd3528c0cab702ebf314e06d3b1e0df0327eb591ea150 not found: ID does not exist" Jul 28 22:59:25.051731 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.051702 2557 scope.go:117] "RemoveContainer" containerID="d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af" Jul 28 22:59:25.055001 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.054978 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgwxd\" (UniqueName: \"kubernetes.io/projected/a8aa3438-7d41-42c7-b7df-cd5792016f96-kube-api-access-dgwxd\") pod \"a8aa3438-7d41-42c7-b7df-cd5792016f96\" (UID: \"a8aa3438-7d41-42c7-b7df-cd5792016f96\") " Jul 28 22:59:25.055108 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.055091 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krs45\" (UniqueName: \"kubernetes.io/projected/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-kube-api-access-krs45\") pod \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\" (UID: \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\") " Jul 28 22:59:25.055177 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.055128 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa3438-7d41-42c7-b7df-cd5792016f96-plugins-config-volume\") pod \"a8aa3438-7d41-42c7-b7df-cd5792016f96\" (UID: \"a8aa3438-7d41-42c7-b7df-cd5792016f96\") " Jul 28 22:59:25.055177 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.055158 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-plugins-config-volume\") pod \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\" (UID: \"41869f0d-b3dd-40c8-b124-a6ee3e2015e1\") " Jul 28 22:59:25.055690 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.055663 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8aa3438-7d41-42c7-b7df-cd5792016f96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8aa3438-7d41-42c7-b7df-cd5792016f96" (UID: "a8aa3438-7d41-42c7-b7df-cd5792016f96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:59:25.055811 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.055724 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41869f0d-b3dd-40c8-b124-a6ee3e2015e1" (UID: "41869f0d-b3dd-40c8-b124-a6ee3e2015e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:59:25.057329 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.057303 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8aa3438-7d41-42c7-b7df-cd5792016f96-kube-api-access-dgwxd" (OuterVolumeSpecName: "kube-api-access-dgwxd") pod "a8aa3438-7d41-42c7-b7df-cd5792016f96" (UID: "a8aa3438-7d41-42c7-b7df-cd5792016f96"). InnerVolumeSpecName "kube-api-access-dgwxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:59:25.057548 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.057534 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-kube-api-access-krs45" (OuterVolumeSpecName: "kube-api-access-krs45") pod "41869f0d-b3dd-40c8-b124-a6ee3e2015e1" (UID: "41869f0d-b3dd-40c8-b124-a6ee3e2015e1"). InnerVolumeSpecName "kube-api-access-krs45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:59:25.060097 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.060082 2557 scope.go:117] "RemoveContainer" containerID="d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af" Jul 28 22:59:25.060356 ip-10-0-142-225 kubenswrapper[2557]: E0728 22:59:25.060339 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af\": container with ID starting with d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af not found: ID does not exist" containerID="d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af" Jul 28 22:59:25.060419 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.060363 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af"} err="failed to get container status \"d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af\": rpc error: code = NotFound desc = could not find container \"d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af\": container with ID starting with d643cb566e859dacf2f39ebc085e7bc9336a64a6f9c3a3cef81b1e4d986619af not found: ID does not exist" Jul 28 22:59:25.156462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.156425 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krs45\" (UniqueName: \"kubernetes.io/projected/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-kube-api-access-krs45\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:59:25.156462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.156454 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8aa3438-7d41-42c7-b7df-cd5792016f96-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:59:25.156462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.156464 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41869f0d-b3dd-40c8-b124-a6ee3e2015e1-plugins-config-volume\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:59:25.156462 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.156472 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgwxd\" (UniqueName: \"kubernetes.io/projected/a8aa3438-7d41-42c7-b7df-cd5792016f96-kube-api-access-dgwxd\") on node \"ip-10-0-142-225.ec2.internal\" DevicePath \"\"" Jul 28 22:59:25.369711 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.369673 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5"] Jul 28 22:59:25.371357 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.371330 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1661d9-86f879d546-9b4h5"] Jul 28 22:59:25.382029 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.382000 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz"] Jul 28 22:59:25.384545 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:25.384519 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1661d9-7f7bfbb5fb-556pz"] Jul 28 22:59:26.046826 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:26.046748 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41869f0d-b3dd-40c8-b124-a6ee3e2015e1" path="/var/lib/kubelet/pods/41869f0d-b3dd-40c8-b124-a6ee3e2015e1/volumes" Jul 28 22:59:26.047198 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:26.047185 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8aa3438-7d41-42c7-b7df-cd5792016f96" path="/var/lib/kubelet/pods/a8aa3438-7d41-42c7-b7df-cd5792016f96/volumes" Jul 28 22:59:27.595031 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:27.594992 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sffm4"] Jul 28 22:59:38.495224 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:38.495195 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d45996775-v5tsd_c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe/copy-manifests/0.log" Jul 28 22:59:38.508616 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:38.508588 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d45996775-v5tsd_c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe/manager/0.log" Jul 28 22:59:41.326623 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:41.326596 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-sffm4_6c8adc46-ef29-49c7-9319-249f5723a69a/limitador/0.log" Jul 28 22:59:41.926610 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:41.926579 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ck7ks_78223ce8-8144-4324-b0d9-bbc11ae63ccc/discovery/0.log" Jul 28 22:59:42.497843 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:42.497814 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-m7ch8_f207a0bb-d507-481b-934d-04eb809019ec/payload-processing/0.log" Jul 28 22:59:43.080035 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:43.080009 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6zptd_0c67f0f7-35a3-4f78-9399-1fe16b6d3e07/main/0.log" Jul 28 22:59:43.417842 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:43.417751 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-9gvgh_a8333182-2a14-4fcc-b680-f929793f117e/main/0.log" Jul 28 22:59:52.325819 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:52.325787 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-76hsd_e7b82635-0361-44b9-9b97-7f496d2be84c/global-pull-secret-syncer/0.log" Jul 28 22:59:52.508351 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:52.508316 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9jwvn_602706ce-d40c-4ee5-9918-4f9855f11cb4/konnectivity-agent/0.log" Jul 28 22:59:52.599376 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:52.599289 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-225.ec2.internal_0269d25d27c08c2cc14488ccb451dba5/haproxy/0.log" Jul 28 22:59:58.137991 ip-10-0-142-225 kubenswrapper[2557]: I0728 22:59:58.137963 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-sffm4_6c8adc46-ef29-49c7-9319-249f5723a69a/limitador/0.log" Jul 28 23:00:00.026281 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.026204 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6b4e66e-1e5d-4b5e-8fe1-97095194fd69/alertmanager/0.log" Jul 28 23:00:00.067905 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.067873 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6b4e66e-1e5d-4b5e-8fe1-97095194fd69/config-reloader/0.log" Jul 28 23:00:00.112402 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.112362 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6b4e66e-1e5d-4b5e-8fe1-97095194fd69/kube-rbac-proxy-web/0.log" Jul 28 23:00:00.142297 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142265 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch"] Jul 28 23:00:00.142659 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142637 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="153e47e8-767c-40f4-998d-485c92de86c6" containerName="payload-pre-processing" Jul 28 23:00:00.142659 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142655 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="153e47e8-767c-40f4-998d-485c92de86c6" containerName="payload-pre-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142669 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8aa3438-7d41-42c7-b7df-cd5792016f96" containerName="payload-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142677 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8aa3438-7d41-42c7-b7df-cd5792016f96" containerName="payload-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142690 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e132bf2-2a7f-4ca0-aaf5-9d5533631c52" containerName="payload-pre-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142699 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e132bf2-2a7f-4ca0-aaf5-9d5533631c52" containerName="payload-pre-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142715 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3ee3fa-6013-42f2-986e-6b761badf0b3" containerName="payload-pre-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142724 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3ee3fa-6013-42f2-986e-6b761badf0b3" containerName="payload-pre-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142736 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae5d913f-599e-4f98-9dc9-4b63a036a851" containerName="payload-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142743 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae5d913f-599e-4f98-9dc9-4b63a036a851" containerName="payload-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142751 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41869f0d-b3dd-40c8-b124-a6ee3e2015e1" containerName="payload-pre-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142756 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="41869f0d-b3dd-40c8-b124-a6ee3e2015e1" containerName="payload-pre-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142762 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e45d50-4a2d-4643-a6b3-03b7614d1f31" containerName="payload-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142766 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e45d50-4a2d-4643-a6b3-03b7614d1f31" containerName="payload-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142781 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f3e8fc-b06e-4d86-8f01-75c216938337" containerName="payload-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142788 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f3e8fc-b06e-4d86-8f01-75c216938337" containerName="payload-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142796 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18d3719-7633-4a89-ae74-d59abb0177c1" containerName="payload-pre-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142803 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18d3719-7633-4a89-ae74-d59abb0177c1" containerName="payload-pre-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142814 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b644adf-f27a-4ec4-9cde-10d658501345" containerName="payload-processing" Jul 28 23:00:00.142827 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142822 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b644adf-f27a-4ec4-9cde-10d658501345" containerName="payload-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142917 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e3ee3fa-6013-42f2-986e-6b761badf0b3" containerName="payload-pre-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142944 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18d3719-7633-4a89-ae74-d59abb0177c1" containerName="payload-pre-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142954 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="41869f0d-b3dd-40c8-b124-a6ee3e2015e1" containerName="payload-pre-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142961 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b644adf-f27a-4ec4-9cde-10d658501345" containerName="payload-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142969 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e45d50-4a2d-4643-a6b3-03b7614d1f31" containerName="payload-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142977 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae5d913f-599e-4f98-9dc9-4b63a036a851" containerName="payload-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142987 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8aa3438-7d41-42c7-b7df-cd5792016f96" containerName="payload-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.142993 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e132bf2-2a7f-4ca0-aaf5-9d5533631c52" containerName="payload-pre-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.143003 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f3e8fc-b06e-4d86-8f01-75c216938337" containerName="payload-processing" Jul 28 23:00:00.143518 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.143010 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="153e47e8-767c-40f4-998d-485c92de86c6" containerName="payload-pre-processing" Jul 28 23:00:00.146176 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.146161 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch" Jul 28 23:00:00.148706 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.148687 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zfvv\"" Jul 28 23:00:00.154521 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.154500 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch"] Jul 28 23:00:00.156641 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.156621 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch" Jul 28 23:00:00.161900 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.161864 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6b4e66e-1e5d-4b5e-8fe1-97095194fd69/kube-rbac-proxy/0.log" Jul 28 23:00:00.217951 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.217866 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6b4e66e-1e5d-4b5e-8fe1-97095194fd69/kube-rbac-proxy-metric/0.log" Jul 28 23:00:00.255801 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.255764 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6b4e66e-1e5d-4b5e-8fe1-97095194fd69/prom-label-proxy/0.log" Jul 28 23:00:00.287418 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.285586 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch"] Jul 28 23:00:00.288560 ip-10-0-142-225 kubenswrapper[2557]: W0728 23:00:00.288538 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b4c130_0698_4057_ad17_cc4bf4c49664.slice/crio-c30897b1d3588ddaf39b11424e965340ef044579d92b763bc8b328bf255e7f8f WatchSource:0}: Error finding container c30897b1d3588ddaf39b11424e965340ef044579d92b763bc8b328bf255e7f8f: Status 404 returned error can't find the container with id c30897b1d3588ddaf39b11424e965340ef044579d92b763bc8b328bf255e7f8f Jul 28 23:00:00.290950 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.290919 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 23:00:00.293649 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.293629 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b6b4e66e-1e5d-4b5e-8fe1-97095194fd69/init-config-reloader/0.log" Jul 28 23:00:00.402998 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.402975 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-hnp4w_704c6d24-83d0-421d-a2a2-d1d33be6a108/kube-state-metrics/0.log" Jul 28 23:00:00.432113 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.432084 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-hnp4w_704c6d24-83d0-421d-a2a2-d1d33be6a108/kube-rbac-proxy-main/0.log" Jul 28 23:00:00.455965 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.455947 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-hnp4w_704c6d24-83d0-421d-a2a2-d1d33be6a108/kube-rbac-proxy-self/0.log" Jul 28 23:00:00.485058 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.485024 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bc7996b9-g9q79_01cfad16-d765-4b87-9ea4-8fdc52705c50/metrics-server/0.log" Jul 28 23:00:00.513497 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.513478 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-5g7v4_3d0f08eb-98cc-47f6-b083-9a2bebbbbaa7/monitoring-plugin/0.log" Jul 28 23:00:00.704714 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.704687 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqkz6_885e99b9-f1b7-4f03-a8a6-0f005b08bf34/node-exporter/0.log" Jul 28 23:00:00.726774 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.726732 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqkz6_885e99b9-f1b7-4f03-a8a6-0f005b08bf34/kube-rbac-proxy/0.log" Jul 28 23:00:00.749798 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.749773 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqkz6_885e99b9-f1b7-4f03-a8a6-0f005b08bf34/init-textfile/0.log" Jul 28 23:00:00.775040 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.775016 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-ts6db_4f94d17c-8df6-4982-b6a7-f2b2d2711769/kube-rbac-proxy-main/0.log" Jul 28 23:00:00.796883 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.796858 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-ts6db_4f94d17c-8df6-4982-b6a7-f2b2d2711769/kube-rbac-proxy-self/0.log" Jul 28 23:00:00.817993 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.817967 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-ts6db_4f94d17c-8df6-4982-b6a7-f2b2d2711769/openshift-state-metrics/0.log" Jul 28 23:00:00.859774 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.859694 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06c336d7-bb1f-4b8f-9a0f-fd6118931a14/prometheus/0.log" Jul 28 23:00:00.883340 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.883319 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06c336d7-bb1f-4b8f-9a0f-fd6118931a14/config-reloader/0.log" Jul 28 23:00:00.905262 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.905241 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06c336d7-bb1f-4b8f-9a0f-fd6118931a14/thanos-sidecar/0.log" Jul 28 23:00:00.929225 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.929201 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06c336d7-bb1f-4b8f-9a0f-fd6118931a14/kube-rbac-proxy-web/0.log" Jul 28 23:00:00.950332 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.950314 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06c336d7-bb1f-4b8f-9a0f-fd6118931a14/kube-rbac-proxy/0.log" Jul 28 23:00:00.971476 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.971460 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06c336d7-bb1f-4b8f-9a0f-fd6118931a14/kube-rbac-proxy-thanos/0.log" Jul 28 23:00:00.993152 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:00.993135 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06c336d7-bb1f-4b8f-9a0f-fd6118931a14/init-config-reloader/0.log" Jul 28 23:00:01.020777 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.020744 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6xc8m_c10b52f5-59ed-4e92-a868-f3844534e1e0/prometheus-operator/0.log" Jul 28 23:00:01.040060 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.040022 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6xc8m_c10b52f5-59ed-4e92-a868-f3844534e1e0/kube-rbac-proxy/0.log" Jul 28 23:00:01.063058 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.063027 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-pf6mx_538621a7-527c-4606-88e0-95bd3c4c4042/prometheus-operator-admission-webhook/0.log" Jul 28 23:00:01.214738 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.214710 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f96b9bf6-fjq27_9b2106be-fee7-49ef-9fff-455c07ab8185/thanos-query/0.log" Jul 28 23:00:01.221588 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.221555 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch" event={"ID":"03b4c130-0698-4057-ad17-cc4bf4c49664","Type":"ContainerStarted","Data":"c30897b1d3588ddaf39b11424e965340ef044579d92b763bc8b328bf255e7f8f"} Jul 28 23:00:01.291012 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.290981 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f96b9bf6-fjq27_9b2106be-fee7-49ef-9fff-455c07ab8185/kube-rbac-proxy-web/0.log" Jul 28 23:00:01.316754 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.316688 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f96b9bf6-fjq27_9b2106be-fee7-49ef-9fff-455c07ab8185/kube-rbac-proxy/0.log" Jul 28 23:00:01.343141 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.343112 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f96b9bf6-fjq27_9b2106be-fee7-49ef-9fff-455c07ab8185/prom-label-proxy/0.log" Jul 28 23:00:01.366578 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.366514 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f96b9bf6-fjq27_9b2106be-fee7-49ef-9fff-455c07ab8185/kube-rbac-proxy-rules/0.log" Jul 28 23:00:01.391248 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:01.391203 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f96b9bf6-fjq27_9b2106be-fee7-49ef-9fff-455c07ab8185/kube-rbac-proxy-metrics/0.log" Jul 28 23:00:03.034625 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.034543 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m"] Jul 28 23:00:03.038130 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.038097 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.040549 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.040530 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lskt7\"/\"kube-root-ca.crt\"" Jul 28 23:00:03.041596 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.041575 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lskt7\"/\"default-dockercfg-6vcgc\"" Jul 28 23:00:03.041705 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.041672 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lskt7\"/\"openshift-service-ca.crt\"" Jul 28 23:00:03.045849 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.045828 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m"] Jul 28 23:00:03.116841 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.116808 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-podres\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.117089 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.116849 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-proc\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.117089 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.116875 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-lib-modules\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.117089 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.116900 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4hw\" (UniqueName: \"kubernetes.io/projected/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-kube-api-access-dc4hw\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.117089 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.117002 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-sys\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.217692 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.217661 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-sys\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.217871 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.217731 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-podres\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.217871 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.217761 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-proc\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.217871 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.217784 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-lib-modules\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.217871 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.217791 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-sys\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.217871 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.217808 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4hw\" (UniqueName: \"kubernetes.io/projected/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-kube-api-access-dc4hw\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.217871 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.217843 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-podres\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.218115 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.217875 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-proc\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.218115 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.217892 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-lib-modules\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.226577 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.226556 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4hw\" (UniqueName: \"kubernetes.io/projected/3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb-kube-api-access-dc4hw\") pod \"perf-node-gather-daemonset-kks4m\" (UID: \"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb\") " pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.233781 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.233754 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch" event={"ID":"03b4c130-0698-4057-ad17-cc4bf4c49664","Type":"ContainerStarted","Data":"c51dd739c10a69258f75d2a59bbf751f3700403d9efe6ea11a2965571804d02d"} Jul 28 23:00:03.248796 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.248749 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch" podStartSLOduration=1.0766476 podStartE2EDuration="3.248732979s" totalImagesPullingTime="2.172085379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:03.247806573 +0000 UTC m=+3327.885665701" watchObservedRunningTime="2026-07-28 23:00:03.248732979 +0000 UTC m=+3327.886592108" Jul 28 23:00:03.349292 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.349266 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:03.479973 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:03.479944 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m"] Jul 28 23:00:03.481683 ip-10-0-142-225 kubenswrapper[2557]: W0728 23:00:03.481650 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3c1fd7db_5f56_4d2e_8d17_ee7f80c773cb.slice/crio-35b964f3f39b2130e0d3e6fec6d8ee523d5824ea4e6ce5a294f643c3d5571f50 WatchSource:0}: Error finding container 35b964f3f39b2130e0d3e6fec6d8ee523d5824ea4e6ce5a294f643c3d5571f50: Status 404 returned error can't find the container with id 35b964f3f39b2130e0d3e6fec6d8ee523d5824ea4e6ce5a294f643c3d5571f50 Jul 28 23:00:04.239794 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:04.239761 2557 generic.go:356] "Generic (PLEG): container finished" podID="03b4c130-0698-4057-ad17-cc4bf4c49664" containerID="c51dd739c10a69258f75d2a59bbf751f3700403d9efe6ea11a2965571804d02d" exitCode=0 Jul 28 23:00:04.240246 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:04.239858 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch" event={"ID":"03b4c130-0698-4057-ad17-cc4bf4c49664","Type":"ContainerDied","Data":"c51dd739c10a69258f75d2a59bbf751f3700403d9efe6ea11a2965571804d02d"} Jul 28 23:00:04.241270 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:04.241246 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" event={"ID":"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb","Type":"ContainerStarted","Data":"4c3f22997f6a5bfbe77520fb291819890711737fa01de95af1e05a2b44672b89"} Jul 28 23:00:04.241374 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:04.241289 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" event={"ID":"3c1fd7db-5f56-4d2e-8d17-ee7f80c773cb","Type":"ContainerStarted","Data":"35b964f3f39b2130e0d3e6fec6d8ee523d5824ea4e6ce5a294f643c3d5571f50"} Jul 28 23:00:04.241374 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:04.241366 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:04.267840 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:04.267792 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" podStartSLOduration=1.267773226 podStartE2EDuration="1.267773226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:04.266813338 +0000 UTC m=+3328.904672466" watchObservedRunningTime="2026-07-28 23:00:04.267773226 +0000 UTC m=+3328.905632383" Jul 28 23:00:05.389550 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:05.389523 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch" Jul 28 23:00:05.509746 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:05.509652 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k7xl5_b009deee-53a6-4016-ba9f-a05750866a41/dns/0.log" Jul 28 23:00:05.530764 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:05.530736 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k7xl5_b009deee-53a6-4016-ba9f-a05750866a41/kube-rbac-proxy/0.log" Jul 28 23:00:05.554623 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:05.554602 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-65mzx_c4009ba5-5c0a-4aa5-8ad6-f1b30c368c57/dns-node-resolver/0.log" Jul 28 23:00:06.165342 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:06.165315 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5zjn6_f7e5aa99-abf9-4ac5-86a9-a34dc7d6f470/node-ca/0.log" Jul 28 23:00:06.253180 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:06.253150 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch" Jul 28 23:00:06.253354 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:06.253181 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-rj4ch" event={"ID":"03b4c130-0698-4057-ad17-cc4bf4c49664","Type":"ContainerDied","Data":"c30897b1d3588ddaf39b11424e965340ef044579d92b763bc8b328bf255e7f8f"} Jul 28 23:00:06.253354 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:06.253228 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c30897b1d3588ddaf39b11424e965340ef044579d92b763bc8b328bf255e7f8f" Jul 28 23:00:07.319581 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:07.319550 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ck7ks_78223ce8-8144-4324-b0d9-bbc11ae63ccc/discovery/0.log" Jul 28 23:00:07.521245 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:07.521217 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-m7ch8_f207a0bb-d507-481b-934d-04eb809019ec/payload-processing/0.log" Jul 28 23:00:08.402553 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:08.402524 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zztbv_482beaea-592d-475b-9e38-c1be4dd15959/serve-healthcheck-canary/0.log" Jul 28 23:00:08.888683 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:08.888644 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7cfns_b7faf624-53f7-4110-bf99-9dd1202d493e/kube-rbac-proxy/0.log" Jul 28 23:00:08.914668 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:08.914642 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7cfns_b7faf624-53f7-4110-bf99-9dd1202d493e/exporter/0.log" Jul 28 23:00:08.937445 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:08.937423 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7cfns_b7faf624-53f7-4110-bf99-9dd1202d493e/extractor/0.log" Jul 28 23:00:10.257106 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:10.257058 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lskt7/perf-node-gather-daemonset-kks4m" Jul 28 23:00:12.166802 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:12.166775 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d45996775-v5tsd_c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe/manager/0.log" Jul 28 23:00:12.185866 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:12.185843 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d45996775-v5tsd_c0acb1a8-d4ca-47a6-8d77-ec6dfa6bcdfe/copy-manifests/0.log" Jul 28 23:00:12.517141 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:12.517111 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ck7ks_78223ce8-8144-4324-b0d9-bbc11ae63ccc/discovery/0.log" Jul 28 23:00:12.664256 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:12.664180 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-m7ch8_f207a0bb-d507-481b-934d-04eb809019ec/payload-processing/0.log" Jul 28 23:00:13.588134 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:13.588107 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-64ddd775c9-5rjc9_3c28ab4e-2288-4075-a2d4-603cc129f2f2/manager/0.log" Jul 28 23:00:14.320862 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:14.320829 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ck7ks_78223ce8-8144-4324-b0d9-bbc11ae63ccc/discovery/0.log" Jul 28 23:00:14.469515 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:14.469485 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-m7ch8_f207a0bb-d507-481b-934d-04eb809019ec/payload-processing/0.log" Jul 28 23:00:19.879322 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:19.879290 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5qcq_5137a1e9-ebea-49c3-9d66-9f836d3bb074/kube-multus-additional-cni-plugins/0.log" Jul 28 23:00:19.907292 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:19.907265 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5qcq_5137a1e9-ebea-49c3-9d66-9f836d3bb074/egress-router-binary-copy/0.log" Jul 28 23:00:19.934717 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:19.934693 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5qcq_5137a1e9-ebea-49c3-9d66-9f836d3bb074/cni-plugins/0.log" Jul 28 23:00:19.957123 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:19.957102 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5qcq_5137a1e9-ebea-49c3-9d66-9f836d3bb074/bond-cni-plugin/0.log" Jul 28 23:00:19.980690 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:19.980669 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5qcq_5137a1e9-ebea-49c3-9d66-9f836d3bb074/routeoverride-cni/0.log" Jul 28 23:00:20.023753 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:20.023727 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5qcq_5137a1e9-ebea-49c3-9d66-9f836d3bb074/whereabouts-cni-bincopy/0.log" Jul 28 23:00:20.046661 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:20.046636 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5qcq_5137a1e9-ebea-49c3-9d66-9f836d3bb074/whereabouts-cni/0.log" Jul 28 23:00:20.389094 ip-10-0-142-225 kubenswrapper[2557]: I0728 23:00:20.389067 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-slckt_95a72516-dfc4-450e-a5db-a3148727c75a/kube-multus/0.log"