Jul 24 12:53:02.586888 ip-10-0-134-100 systemd[1]: Starting Kubernetes Kubelet... Jul 24 12:53:03.003849 ip-10-0-134-100 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 12:53:03.003849 ip-10-0-134-100 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 12:53:03.003849 ip-10-0-134-100 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 12:53:03.003849 ip-10-0-134-100 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 12:53:03.003849 ip-10-0-134-100 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 12:53:03.004849 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.004759 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 12:53:03.009854 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009831 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 12:53:03.009854 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009850 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 12:53:03.009854 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009855 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 12:53:03.009854 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009859 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009863 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009867 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009871 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009874 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009878 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009883 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009887 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009891 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009894 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009898 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009902 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009906 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009910 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009914 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009919 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009922 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009927 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009931 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009934 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 12:53:03.010103 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009938 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009942 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009946 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009950 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009954 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009967 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009974 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009980 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009984 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009991 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.009997 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010003 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010007 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010013 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010018 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010023 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010029 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010033 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010037 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 12:53:03.010979 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010042 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010046 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010050 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010055 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010059 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010063 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010068 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010072 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010077 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010081 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010085 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010089 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010093 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010098 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010101 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010105 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010109 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010113 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010117 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010121 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 12:53:03.011882 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010125 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010132 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010139 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010143 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010148 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010152 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010157 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010161 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010165 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010169 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010173 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010178 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010183 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010187 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010191 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010195 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010199 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010204 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010209 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 12:53:03.012767 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010213 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010220 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010224 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010229 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010233 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010237 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010241 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010245 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010249 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010253 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010257 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010262 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010266 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010270 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010274 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010279 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010283 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.010288 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011505 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011516 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 12:53:03.013633 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011521 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011525 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011530 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011533 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011537 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011542 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011546 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011550 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011554 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011558 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011563 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011567 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011572 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011596 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011601 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011605 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011609 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011613 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011617 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011622 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 12:53:03.014505 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011629 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011635 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011640 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011645 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011651 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011655 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011659 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011664 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011668 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011672 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011676 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011680 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011684 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011689 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011693 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011697 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011701 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011705 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011709 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011713 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 12:53:03.015233 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011717 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011721 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011725 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011730 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011734 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011739 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011744 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011748 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011752 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011756 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011760 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011764 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011768 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011772 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011776 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011780 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011786 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011790 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011794 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011798 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 12:53:03.016020 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011802 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011806 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011811 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011815 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011819 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011823 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011827 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011831 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011835 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011841 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011845 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011850 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011854 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011858 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011862 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011866 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011870 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011877 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011883 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 12:53:03.016668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011888 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011892 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011897 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011901 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011905 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011913 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011918 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011924 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011929 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011934 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011939 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011943 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011947 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011951 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011955 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011960 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011964 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.011969 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012680 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 12:53:03.017168 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012691 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012696 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012701 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012705 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012709 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012714 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012719 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012723 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012727 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012731 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012735 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012739 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012748 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012753 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012757 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012761 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012766 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012770 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012774 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 12:53:03.017883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012778 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012782 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012787 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012791 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012795 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012799 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012803 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012807 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012812 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012816 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012820 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012824 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012828 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012832 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012836 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012840 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012844 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012848 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012852 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012859 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 12:53:03.018374 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012863 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012867 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012872 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012876 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012880 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012885 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012898 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012903 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012907 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012912 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012916 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012920 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012924 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012928 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012932 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012936 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012941 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012945 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012949 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012953 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 12:53:03.018935 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012959 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012964 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012969 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012974 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012979 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012985 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012989 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012994 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.012999 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013002 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013006 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013011 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013014 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013019 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013025 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013031 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013035 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013042 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013046 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 12:53:03.019489 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013059 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013064 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013068 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013072 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013076 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013080 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013085 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013089 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013095 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013101 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013105 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013110 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013115 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013119 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013123 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013127 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013132 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013135 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013140 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.013144 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 12:53:03.019993 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013249 2554 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013259 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013275 2554 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013283 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013290 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013295 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013301 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013308 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013312 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013317 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013326 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013331 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013336 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013357 2554 flags.go:64] FLAG: --cgroup-root="" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013363 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013368 2554 flags.go:64] FLAG: --client-ca-file="" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013373 2554 flags.go:64] FLAG: --cloud-provider="external" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013377 2554 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013386 2554 flags.go:64] FLAG: --cluster-domain="" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013391 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013396 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013401 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013407 2554 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013418 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 12:53:03.020597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013423 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013427 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013433 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013437 2554 flags.go:64] FLAG: --contention-profiling="false" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013442 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013447 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013452 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013456 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013463 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013468 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013474 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013478 2554 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013483 2554 flags.go:64] FLAG: --enable-server="true" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013488 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013495 2554 flags.go:64] FLAG: --event-burst="100" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013501 2554 flags.go:64] FLAG: --event-qps="50" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013505 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013510 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013516 2554 flags.go:64] FLAG: --eviction-hard="" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013522 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013527 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013531 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013544 2554 flags.go:64] FLAG: --eviction-soft="" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013549 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013553 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 12:53:03.021250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013558 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013562 2554 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013567 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013571 2554 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013595 2554 flags.go:64] FLAG: --feature-gates="" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013603 2554 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013607 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013612 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013617 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013622 2554 flags.go:64] FLAG: --healthz-port="10248" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013627 2554 flags.go:64] FLAG: --help="false" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013631 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013636 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013640 2554 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013645 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013650 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013657 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013662 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013666 2554 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013670 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013675 2554 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013680 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013685 2554 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013690 2554 flags.go:64] FLAG: --kube-reserved="" Jul 24 12:53:03.021937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013694 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013699 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013704 2554 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013709 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013713 2554 flags.go:64] FLAG: --lock-file="" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013718 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013731 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013736 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013745 2554 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013749 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013754 2554 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013758 2554 flags.go:64] FLAG: --logging-format="text" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013763 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013768 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013773 2554 flags.go:64] FLAG: --manifest-url="" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013777 2554 flags.go:64] FLAG: --manifest-url-header="" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013784 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013789 2554 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013795 2554 flags.go:64] FLAG: --max-pods="110" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013800 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013804 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013809 2554 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013814 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013818 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013825 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 12:53:03.022567 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013829 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013840 2554 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013845 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013850 2554 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013855 2554 flags.go:64] FLAG: --pod-cidr="" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013859 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013867 2554 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013872 2554 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013876 2554 flags.go:64] FLAG: --pods-per-core="0" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013881 2554 flags.go:64] FLAG: --port="10250" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013886 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013891 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-079596ec996b24479" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013896 2554 flags.go:64] FLAG: --qos-reserved="" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013901 2554 flags.go:64] FLAG: --read-only-port="10255" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013914 2554 flags.go:64] FLAG: --register-node="true" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013918 2554 flags.go:64] FLAG: --register-with-taints="" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013924 2554 flags.go:64] FLAG: --registry-burst="10" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013929 2554 flags.go:64] FLAG: --registry-qps="5" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013935 2554 flags.go:64] FLAG: --reserved-cpus="" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013939 2554 flags.go:64] FLAG: --reserved-memory="" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013946 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013951 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013955 2554 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013960 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013965 2554 flags.go:64] FLAG: --runonce="false" Jul 24 12:53:03.023242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013969 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013974 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013979 2554 flags.go:64] FLAG: --seccomp-default="false" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013984 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013988 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013993 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.013999 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014004 2554 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014008 2554 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014013 2554 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014018 2554 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014022 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014027 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014032 2554 flags.go:64] FLAG: --system-cgroups="" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014037 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014045 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014050 2554 flags.go:64] FLAG: --tls-cert-file="" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014054 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014061 2554 flags.go:64] FLAG: --tls-min-version="" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014065 2554 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014069 2554 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014074 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014087 2554 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014092 2554 flags.go:64] FLAG: --v="2" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014098 2554 flags.go:64] FLAG: --version="false" Jul 24 12:53:03.023915 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014105 2554 flags.go:64] FLAG: --vmodule="" Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014111 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014117 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014303 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014310 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014316 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014321 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014325 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014330 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014334 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014338 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014342 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014346 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014353 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014357 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014361 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014365 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014369 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014373 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014377 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 12:53:03.024616 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014381 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014385 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014389 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014393 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014397 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014401 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014405 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014409 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014413 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014417 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014422 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014426 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014430 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014434 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014438 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014442 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014446 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014451 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014454 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014458 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 12:53:03.025149 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014462 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014466 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014470 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014474 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014478 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014483 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014487 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014491 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014495 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014499 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014503 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014508 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014512 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014516 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014520 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014524 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014529 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014536 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014541 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 12:53:03.025692 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014546 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014551 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014556 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014560 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014566 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014570 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014594 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014602 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014607 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014611 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014618 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014623 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014628 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014633 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014637 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014642 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014646 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014650 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014656 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 12:53:03.026174 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014660 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014664 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014668 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014673 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014677 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014683 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014687 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014691 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014695 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014700 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014704 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014708 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014713 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014717 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014721 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014725 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014729 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014733 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014739 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014744 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 12:53:03.026689 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014748 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014752 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014763 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.014767 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.014781 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.022003 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.022018 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.022038 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.022044 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022096 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022101 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022105 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022108 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022112 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 12:53:03.027222 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022116 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022119 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022122 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022125 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022128 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022130 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022132 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022135 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022137 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022140 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022143 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022145 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022148 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022150 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022153 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022155 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022158 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022161 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022163 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022166 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 12:53:03.027603 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022168 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022171 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022173 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022176 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022183 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022186 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022189 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022191 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022194 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022196 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022199 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022201 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022204 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022206 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022208 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022211 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022213 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022216 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022218 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022220 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 12:53:03.028129 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022223 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022225 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022228 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022232 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022235 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022238 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022241 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022243 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022246 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022248 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022250 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022253 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022255 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022258 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022260 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022263 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022266 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022269 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022271 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022274 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 12:53:03.028673 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022276 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022278 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022281 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022283 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022287 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022291 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022294 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022297 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022299 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022302 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022305 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022307 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022309 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022312 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022314 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022317 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022319 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022321 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022324 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022326 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 12:53:03.029204 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022328 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022330 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022333 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022335 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022338 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022340 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022343 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022345 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022348 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022351 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022354 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022356 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022359 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022361 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.022366 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 12:53:03.029740 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022495 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022500 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022503 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022507 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022510 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022513 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022516 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022518 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022521 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022524 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022527 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022529 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022532 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022534 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022537 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022539 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022541 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022544 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022546 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022548 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 12:53:03.030147 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022551 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022553 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022556 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022558 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022561 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022564 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022566 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022569 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022572 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022575 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022596 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022600 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022604 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022608 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022611 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022614 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022618 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022621 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022624 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 12:53:03.030677 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022626 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022629 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022631 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022634 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022636 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022638 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022641 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022643 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022646 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022649 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022651 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022654 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022656 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022659 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022661 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022664 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022666 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022669 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022672 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022675 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 12:53:03.031180 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022678 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022680 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022684 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022687 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022689 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022691 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022694 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022696 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022699 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022701 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022704 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022715 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022718 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022721 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022724 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022727 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022730 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022733 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022736 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022738 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 12:53:03.031710 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022741 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022743 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022747 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022750 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022753 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022762 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022766 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022769 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022771 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022774 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022777 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022780 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022783 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022786 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022789 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022792 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022795 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022797 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022800 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 12:53:03.032213 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:03.022802 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 12:53:03.032711 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.022807 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 12:53:03.032711 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.022951 2554 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 12:53:03.032711 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.025748 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 12:53:03.032711 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.027076 2554 server.go:1019] "Starting client certificate rotation" Jul 24 12:53:03.032711 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.027177 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 12:53:03.032711 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.027211 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 12:53:03.032711 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.030366 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 12:53:03.033937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.033921 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 12:53:03.046041 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.046021 2554 log.go:25] "Validated CRI v1 runtime API" Jul 24 12:53:03.048100 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.048080 2554 log.go:25] "Validated CRI v1 image API" Jul 24 12:53:03.048955 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.048937 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 12:53:03.054438 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.054411 2554 fs.go:135] Filesystem UUIDs: map[36f2a994-c586-48f6-acbf-a611d56dd42d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b8bdb907-2f30-4a88-bc1f-a5bc2d81f741:/dev/nvme0n1p4] Jul 24 12:53:03.054517 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.054438 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 12:53:03.059804 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.059669 2554 manager.go:216] Machine: {Timestamp:2026-07-24 12:53:03.05833423 +0000 UTC m=+0.360615853 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099105 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297fc66c7536d8b89d2753713bf815 SystemUUID:ec297fc6-6c75-36d8-b89d-2753713bf815 BootID:3e55548b-8e9b-4c39-a362-404a1193b0b4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:83:fc:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:83:fc:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:17:81:a1:38:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 12:53:03.059804 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.059795 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 12:53:03.059930 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.059873 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 12:53:03.061201 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.061184 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 12:53:03.061901 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.061873 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 12:53:03.062044 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.061904 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 12:53:03.062534 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.062524 2554 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 12:53:03.062565 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.062537 2554 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 12:53:03.062565 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.062545 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 12:53:03.063360 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.063348 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 12:53:03.064478 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.064468 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 12:53:03.067765 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.067753 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 24 12:53:03.067895 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.067885 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 12:53:03.067928 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.067917 2554 kubelet.go:486] "Attempting to sync node with API server" Jul 24 12:53:03.067978 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.067931 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 12:53:03.067978 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.067953 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 12:53:03.067978 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.067966 2554 kubelet.go:398] "Adding apiserver pod source" Jul 24 12:53:03.068088 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.067990 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 12:53:03.068428 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.068415 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 12:53:03.068484 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.068437 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 12:53:03.070958 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.070944 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 12:53:03.072292 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.072279 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 12:53:03.072345 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.072310 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 12:53:03.072930 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.072920 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 12:53:03.072968 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.072938 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 12:53:03.072968 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.072944 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 12:53:03.072968 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.072949 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 12:53:03.072968 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.072961 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 12:53:03.072968 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.072970 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 12:53:03.073237 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.073229 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 12:53:03.073271 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.073238 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 12:53:03.073271 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.073244 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 12:53:03.073271 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.073249 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 12:53:03.073271 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.073257 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 12:53:03.073271 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.073267 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 12:53:03.074000 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.073990 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 12:53:03.074038 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.074002 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 12:53:03.076449 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.076432 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 12:53:03.076544 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.076527 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 12:53:03.076600 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.076522 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 12:53:03.077301 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.077289 2554 server.go:1268] "Started kubelet" Jul 24 12:53:03.077411 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.077343 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 12:53:03.077470 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.077387 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 12:53:03.077470 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.077450 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 12:53:03.078193 ip-10-0-134-100 systemd[1]: Started Kubernetes Kubelet. Jul 24 12:53:03.078994 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.078980 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 12:53:03.084062 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.081822 2554 server.go:310] "Adding debug handlers to kubelet server" Jul 24 12:53:03.085377 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.085359 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 12:53:03.087053 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.086021 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-100.ec2.internal.18c53ae51fc6049a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-100.ec2.internal,UID:ip-10-0-134-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-100.ec2.internal,},FirstTimestamp:2026-07-24 12:53:03.07726249 +0000 UTC m=+0.379544114,LastTimestamp:2026-07-24 12:53:03.07726249 +0000 UTC m=+0.379544114,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-100.ec2.internal,}" Jul 24 12:53:03.087555 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.087539 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f756l" Jul 24 12:53:03.089744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.089728 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 12:53:03.089744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.089738 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 12:53:03.089967 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.089947 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:03.090292 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090278 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 12:53:03.090419 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090396 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 12:53:03.090419 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090278 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 12:53:03.090593 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090472 2554 reconstruct.go:97] "Volume reconstruction finished" Jul 24 12:53:03.090593 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090481 2554 reconciler.go:29] "Reconciler: start to sync state" Jul 24 12:53:03.090866 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090738 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 12:53:03.090866 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090749 2554 factory.go:55] Registering systemd factory Jul 24 12:53:03.090866 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090766 2554 factory.go:223] Registration of the systemd container factory successfully Jul 24 12:53:03.090993 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090970 2554 factory.go:153] Registering CRI-O factory Jul 24 12:53:03.090993 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.090983 2554 factory.go:223] Registration of the crio container factory successfully Jul 24 12:53:03.091078 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.091008 2554 factory.go:103] Registering Raw factory Jul 24 12:53:03.091078 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.091023 2554 manager.go:1196] Started watching for new ooms in manager Jul 24 12:53:03.091403 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.091389 2554 manager.go:349] Starting recovery of all containers Jul 24 12:53:03.096934 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.096912 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f756l" Jul 24 12:53:03.098417 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.098385 2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 12:53:03.098517 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.098495 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 12:53:03.103329 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.103314 2554 manager.go:354] Recovery completed Jul 24 12:53:03.108461 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.108449 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 12:53:03.110822 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.110807 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 12:53:03.110877 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.110837 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 12:53:03.110877 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.110848 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 12:53:03.111370 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.111358 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 12:53:03.111370 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.111369 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 12:53:03.111484 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.111375 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 12:53:03.111484 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.111384 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 24 12:53:03.112093 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.112076 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 12:53:03.112180 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.112109 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 12:53:03.112180 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.112129 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 12:53:03.112503 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.112492 2554 policy_none.go:49] "None policy: Start" Jul 24 12:53:03.112536 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.112508 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 12:53:03.112570 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.112540 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 12:53:03.113109 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.113039 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-100.ec2.internal.18c53ae521c61b66 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-100.ec2.internal,UID:ip-10-0-134-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-100.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-100.ec2.internal,},FirstTimestamp:2026-07-24 12:53:03.110822758 +0000 UTC m=+0.413104382,LastTimestamp:2026-07-24 12:53:03.110822758 +0000 UTC m=+0.413104382,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-100.ec2.internal,}" Jul 24 12:53:03.113548 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.113536 2554 policy_none.go:47] "Start" Jul 24 12:53:03.149744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.148755 2554 manager.go:337] "Starting Device Plugin manager" Jul 24 12:53:03.149744 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.148962 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 12:53:03.149744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.148974 2554 server.go:87] "Starting device plugin registration server" Jul 24 12:53:03.149744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.149237 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 12:53:03.149744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.149249 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 12:53:03.149744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.149575 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 12:53:03.149744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.149707 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 12:53:03.149744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.149720 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 12:53:03.150684 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.150627 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 12:53:03.150787 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.150686 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:03.249911 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.249883 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 12:53:03.251020 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.250979 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 12:53:03.251020 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.251011 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 12:53:03.251020 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.251022 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 12:53:03.251152 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.251046 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.260007 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.259956 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.260007 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.259982 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-100.ec2.internal\": node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:03.274128 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.274112 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:03.291958 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.291931 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 12:53:03.293309 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.293288 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 12:53:03.293383 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.293324 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 12:53:03.293383 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.293348 2554 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 12:53:03.293467 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.293388 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 12:53:03.295392 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.295373 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 12:53:03.374266 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.374231 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:03.393706 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.393679 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal"] Jul 24 12:53:03.393773 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.393751 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 12:53:03.394913 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.394895 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 12:53:03.395023 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.394933 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 12:53:03.395023 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.394949 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 12:53:03.396444 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.396429 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 12:53:03.396597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.396555 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.396652 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.396611 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 12:53:03.398952 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.398935 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 12:53:03.399020 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.398941 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 12:53:03.399020 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.398985 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 12:53:03.399020 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.398994 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 12:53:03.399020 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.398962 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 12:53:03.399145 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.399042 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 12:53:03.400253 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.400237 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.400310 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.400266 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 12:53:03.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.400917 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 12:53:03.400997 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.400946 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 12:53:03.400997 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.400957 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 12:53:03.424268 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.424245 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-100.ec2.internal\" not found" node="ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.428691 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.428676 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-100.ec2.internal\" not found" node="ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.474354 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.474332 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:03.492123 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.492095 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aeed6dc75c412f2f51097cace09f376d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal\" (UID: \"aeed6dc75c412f2f51097cace09f376d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.492123 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.492122 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aeed6dc75c412f2f51097cace09f376d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal\" (UID: \"aeed6dc75c412f2f51097cace09f376d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.492263 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.492138 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7de245839c2e01ba31555968cc0536f8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-100.ec2.internal\" (UID: \"7de245839c2e01ba31555968cc0536f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.574748 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.574663 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:03.593051 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.593010 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aeed6dc75c412f2f51097cace09f376d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal\" (UID: \"aeed6dc75c412f2f51097cace09f376d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.593051 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.593034 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aeed6dc75c412f2f51097cace09f376d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal\" (UID: \"aeed6dc75c412f2f51097cace09f376d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.593179 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.593075 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aeed6dc75c412f2f51097cace09f376d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal\" (UID: \"aeed6dc75c412f2f51097cace09f376d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.593179 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.593100 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7de245839c2e01ba31555968cc0536f8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-100.ec2.internal\" (UID: \"7de245839c2e01ba31555968cc0536f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.593179 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.593116 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aeed6dc75c412f2f51097cace09f376d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal\" (UID: \"aeed6dc75c412f2f51097cace09f376d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.593179 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.593135 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7de245839c2e01ba31555968cc0536f8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-100.ec2.internal\" (UID: \"7de245839c2e01ba31555968cc0536f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.675196 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.675150 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:03.726764 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.726734 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.731897 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:03.731879 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal" Jul 24 12:53:03.776005 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.775978 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:03.876680 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:03.876565 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:05.276055 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:05.276018 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:05.325626 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.325594 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 12:53:05.326068 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.325746 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 12:53:05.376164 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:05.376128 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:05.401406 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.401358 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 12:48:04 +0000 UTC" deadline="2028-01-25 18:28:19.044462601 +0000 UTC" Jul 24 12:53:05.401406 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.401395 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13205h35m13.643070068s" Jul 24 12:53:05.461687 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.461658 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 12:53:05.476262 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:05.476211 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:05.546171 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:05.546142 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de245839c2e01ba31555968cc0536f8.slice/crio-932aba8cf45b14f3a269b41ffe175ec9be9e2b13a71092b3f8cac4d59fc5163b WatchSource:0}: Error finding container 932aba8cf45b14f3a269b41ffe175ec9be9e2b13a71092b3f8cac4d59fc5163b: Status 404 returned error can't find the container with id 932aba8cf45b14f3a269b41ffe175ec9be9e2b13a71092b3f8cac4d59fc5163b Jul 24 12:53:05.546531 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:05.546518 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeed6dc75c412f2f51097cace09f376d.slice/crio-74782d967d3b20c1ee4b66dd59b12736c0345f9e16ff5710c2ded68273011dec WatchSource:0}: Error finding container 74782d967d3b20c1ee4b66dd59b12736c0345f9e16ff5710c2ded68273011dec: Status 404 returned error can't find the container with id 74782d967d3b20c1ee4b66dd59b12736c0345f9e16ff5710c2ded68273011dec Jul 24 12:53:05.552645 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.552629 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 12:53:05.576455 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:05.576425 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:05.595933 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.595883 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal" event={"ID":"7de245839c2e01ba31555968cc0536f8","Type":"ContainerStarted","Data":"932aba8cf45b14f3a269b41ffe175ec9be9e2b13a71092b3f8cac4d59fc5163b"} Jul 24 12:53:05.596880 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.596850 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" event={"ID":"aeed6dc75c412f2f51097cace09f376d","Type":"ContainerStarted","Data":"74782d967d3b20c1ee4b66dd59b12736c0345f9e16ff5710c2ded68273011dec"} Jul 24 12:53:05.677141 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:05.677099 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:05.750530 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.750450 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 12:53:05.778046 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:05.778009 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:05.878952 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:05.878907 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-100.ec2.internal\" not found" Jul 24 12:53:05.884108 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.884089 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 12:53:05.889429 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.889399 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" Jul 24 12:53:05.902610 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.902582 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 12:53:05.903709 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.903691 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal" Jul 24 12:53:05.914072 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:05.914053 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 12:53:06.160289 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.160219 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 12:53:06.368029 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.367994 2554 apiserver.go:52] "Watching apiserver" Jul 24 12:53:06.378890 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.378851 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 12:53:06.379260 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.379238 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl","openshift-dns/node-resolver-99gm8","openshift-multus/multus-vcl2c","openshift-multus/network-metrics-daemon-jcst2","openshift-network-diagnostics/network-check-target-jwxch","openshift-network-operator/iptables-alerter-n4x4g","openshift-ovn-kubernetes/ovnkube-node-jtlpz","kube-system/konnectivity-agent-kpqcc","kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal","openshift-cluster-node-tuning-operator/tuned-zqf8d","openshift-image-registry/node-ca-sx82n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal","openshift-multus/multus-additional-cni-plugins-6ljtj","kube-system/global-pull-secret-syncer-ltm9n"] Jul 24 12:53:06.380908 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.380886 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.382329 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.382312 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.383610 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.383593 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.386876 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.385669 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 12:53:06.386876 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.386158 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 12:53:06.386876 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.386343 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:06.386876 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.386552 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:06.386876 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.386702 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 12:53:06.388722 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.388647 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:06.388826 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.388772 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:06.390490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.388920 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 12:53:06.390490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.389498 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jq85z\"" Jul 24 12:53:06.390490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.389548 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 12:53:06.390490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.389709 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 12:53:06.390490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.390096 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.390490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.390311 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 12:53:06.392227 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.392208 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 12:53:06.393296 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.393279 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gqmxb\"" Jul 24 12:53:06.393412 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.393393 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6mrxj\"" Jul 24 12:53:06.393731 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.393709 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:06.393875 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.393473 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 12:53:06.393875 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.393785 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 12:53:06.393875 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.393503 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 12:53:06.393875 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.393536 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 12:53:06.394097 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.393637 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 12:53:06.394097 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.393786 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:06.395065 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.395047 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.396370 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.396346 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.397224 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.397206 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 12:53:06.397797 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.397526 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 12:53:06.397797 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.397543 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 12:53:06.397934 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.397804 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.399734 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.399714 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 12:53:06.399942 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.399911 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-slash\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.399942 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.399943 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-log-socket\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400091 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.399965 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-cni-bin\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400091 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.399989 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400091 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400016 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6af1e2e-1a39-4433-80a8-563a45ebc689-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.400091 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400041 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x42l\" (UniqueName: \"kubernetes.io/projected/b6af1e2e-1a39-4433-80a8-563a45ebc689-kube-api-access-9x42l\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.400091 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400065 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-os-release\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400091 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400088 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-run-systemd\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400113 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-run-openvswitch\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400137 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-node-log\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400160 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-cnibin\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400183 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-cnibin\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400207 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-run-k8s-cni-cncf-io\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400233 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-var-lib-kubelet\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400256 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-multus-conf-dir\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400281 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-var-lib-openvswitch\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400303 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dd3231a-0da1-44e5-a881-5544caae8e8b-ovnkube-config\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400337 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dd3231a-0da1-44e5-a881-5544caae8e8b-ovnkube-script-lib\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400382 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400363 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb44g\" (UniqueName: \"kubernetes.io/projected/2dd3231a-0da1-44e5-a881-5544caae8e8b-kube-api-access-vb44g\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400404 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-system-cni-dir\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400431 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6af1e2e-1a39-4433-80a8-563a45ebc689-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400454 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-multus-socket-dir-parent\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400489 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-var-lib-cni-bin\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400516 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dd3231a-0da1-44e5-a881-5544caae8e8b-env-overrides\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400540 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91637e1c-200e-429b-ac75-147a1cc1f222-multus-daemon-config\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400588 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83ecea52-7a59-4422-9428-94dfaacff497-host-slash\") pod \"iptables-alerter-n4x4g\" (UID: \"83ecea52-7a59-4422-9428-94dfaacff497\") " pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400617 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hcsk\" (UniqueName: \"kubernetes.io/projected/83ecea52-7a59-4422-9428-94dfaacff497-kube-api-access-2hcsk\") pod \"iptables-alerter-n4x4g\" (UID: \"83ecea52-7a59-4422-9428-94dfaacff497\") " pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400651 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-systemd-units\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400674 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-run-netns\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400699 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-etc-openvswitch\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400731 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-run-multus-certs\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400760 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-etc-kubernetes\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400784 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83ecea52-7a59-4422-9428-94dfaacff497-iptables-alerter-script\") pod \"iptables-alerter-n4x4g\" (UID: \"83ecea52-7a59-4422-9428-94dfaacff497\") " pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400821 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-run-ovn\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.400936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400847 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-cni-netd\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400873 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91637e1c-200e-429b-ac75-147a1cc1f222-cni-binary-copy\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400896 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-run-netns\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400921 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400948 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.400986 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dd3231a-0da1-44e5-a881-5544caae8e8b-ovn-node-metrics-cert\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401013 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-os-release\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401039 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-multus-cni-dir\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401063 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m79bj\" (UniqueName: \"kubernetes.io/projected/91637e1c-200e-429b-ac75-147a1cc1f222-kube-api-access-m79bj\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401087 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401115 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mdkn\" (UniqueName: \"kubernetes.io/projected/415e7d0e-e69a-46d7-b32c-ad60c5487fab-kube-api-access-9mdkn\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401140 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-kubelet\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401167 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401198 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6af1e2e-1a39-4433-80a8-563a45ebc689-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401224 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-system-cni-dir\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401259 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-var-lib-cni-multus\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401284 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-hostroot\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.401762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401492 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 12:53:06.402585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401708 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 12:53:06.402585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401742 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-27ddb\"" Jul 24 12:53:06.402585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401876 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zv69f\"" Jul 24 12:53:06.402585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401876 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.402585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.401951 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vjqbq\"" Jul 24 12:53:06.402585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.402182 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 12:53:06.402585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.402392 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 12:53:06.402585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.402582 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 12:53:06.403092 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.402799 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x7gvx\"" Jul 24 12:53:06.403463 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.403435 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:06.404255 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.404237 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 12:53:06.407030 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.406823 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 12:53:06.407030 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.406874 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 12:53:06.407030 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.406906 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 12:53:06.407030 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.406933 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 12:53:06.407344 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.407179 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4bm2h\"" Jul 24 12:53:06.407344 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.407243 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 12:53:06.407453 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.407428 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bjxlp\"" Jul 24 12:53:06.411775 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.411725 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 12:53:06.445510 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.445477 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lxpll" Jul 24 12:53:06.453407 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.453385 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lxpll" Jul 24 12:53:06.490414 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.490388 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 12:53:06.501775 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501744 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-var-lib-cni-multus\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.501935 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501793 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fxxb\" (UniqueName: \"kubernetes.io/projected/3777315e-9137-4504-b546-c35d39f4613b-kube-api-access-7fxxb\") pod \"node-resolver-99gm8\" (UID: \"3777315e-9137-4504-b546-c35d39f4613b\") " pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.501935 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501822 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83ecea52-7a59-4422-9428-94dfaacff497-host-slash\") pod \"iptables-alerter-n4x4g\" (UID: \"83ecea52-7a59-4422-9428-94dfaacff497\") " pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.501935 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501849 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-slash\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.501935 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501874 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-cni-bin\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.501935 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501873 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-var-lib-cni-multus\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.501935 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501908 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83ecea52-7a59-4422-9428-94dfaacff497-host-slash\") pod \"iptables-alerter-n4x4g\" (UID: \"83ecea52-7a59-4422-9428-94dfaacff497\") " pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.501935 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501917 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-multus-conf-dir\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501954 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-cni-bin\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501962 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-multus-conf-dir\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501927 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-slash\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501962 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-run-multus-certs\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.501994 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-run-multus-certs\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502003 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-run-systemd\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502028 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-run-openvswitch\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502051 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-node-log\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502073 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-var-lib-kubelet\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502097 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5073278-ac92-4fcd-a3bf-236f10e39dad-host\") pod \"node-ca-sx82n\" (UID: \"d5073278-ac92-4fcd-a3bf-236f10e39dad\") " pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502115 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-var-lib-openvswitch\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502122 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-run-openvswitch\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502133 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dd3231a-0da1-44e5-a881-5544caae8e8b-ovnkube-config\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502029 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-run-systemd\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502160 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-var-lib-kubelet\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502176 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-var-lib-openvswitch\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502172 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6af1e2e-1a39-4433-80a8-563a45ebc689-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.502348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502213 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-multus-socket-dir-parent\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502172 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-node-log\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502235 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91637e1c-200e-429b-ac75-147a1cc1f222-multus-daemon-config\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502263 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502285 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-sys\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502295 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-multus-socket-dir-parent\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502308 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dd3231a-0da1-44e5-a881-5544caae8e8b-env-overrides\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502350 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-socket-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502462 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-device-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502498 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbea5b3b-f397-4df0-a942-9b238c3d15aa-dbus\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502531 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-systemd\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502573 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-lib-modules\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502597 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-host\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502624 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-etc-openvswitch\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502648 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-etc-kubernetes\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502699 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-etc-kubernetes\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502701 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-etc-openvswitch\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503105 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502731 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dd3231a-0da1-44e5-a881-5544caae8e8b-env-overrides\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-run-ovn\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502767 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-cni-netd\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502792 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-run-netns\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502797 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dd3231a-0da1-44e5-a881-5544caae8e8b-ovnkube-config\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502812 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6af1e2e-1a39-4433-80a8-563a45ebc689-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502817 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502821 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-cni-netd\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502815 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-run-ovn\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502836 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-run-netns\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502843 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-os-release\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502872 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502898 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdkn\" (UniqueName: \"kubernetes.io/projected/415e7d0e-e69a-46d7-b32c-ad60c5487fab-kube-api-access-9mdkn\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502902 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-os-release\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502923 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3777315e-9137-4504-b546-c35d39f4613b-hosts-file\") pod \"node-resolver-99gm8\" (UID: \"3777315e-9137-4504-b546-c35d39f4613b\") " pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502941 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91637e1c-200e-429b-ac75-147a1cc1f222-multus-daemon-config\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.502944 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckjv5\" (UniqueName: \"kubernetes.io/projected/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-kube-api-access-ckjv5\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.503002 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:06.503858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503067 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-kubelet\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503109 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-kubelet\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.503125 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs podName:415e7d0e-e69a-46d7-b32c-ad60c5487fab nodeName:}" failed. No retries permitted until 2026-07-24 12:53:07.003088656 +0000 UTC m=+3.006390743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs") pod "network-metrics-daemon-jcst2" (UID: "415e7d0e-e69a-46d7-b32c-ad60c5487fab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503147 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503168 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6af1e2e-1a39-4433-80a8-563a45ebc689-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503184 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-system-cni-dir\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503202 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-hostroot\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503238 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-sys-fs\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503259 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-kubernetes\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503262 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-system-cni-dir\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503277 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-log-socket\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503295 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503312 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-hostroot\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503317 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-log-socket\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503316 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6af1e2e-1a39-4433-80a8-563a45ebc689-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503343 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.504654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503341 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503359 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x42l\" (UniqueName: \"kubernetes.io/projected/b6af1e2e-1a39-4433-80a8-563a45ebc689-kube-api-access-9x42l\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503387 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-os-release\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503412 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-var-lib-kubelet\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503451 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83ecea52-7a59-4422-9428-94dfaacff497-iptables-alerter-script\") pod \"iptables-alerter-n4x4g\" (UID: \"83ecea52-7a59-4422-9428-94dfaacff497\") " pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503475 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-cnibin\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503475 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-os-release\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503500 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-cnibin\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503522 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-run-k8s-cni-cncf-io\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503546 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbea5b3b-f397-4df0-a942-9b238c3d15aa-kubelet-config\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503554 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-cnibin\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503598 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-tuned\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503624 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dd3231a-0da1-44e5-a881-5544caae8e8b-ovnkube-script-lib\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503651 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb44g\" (UniqueName: \"kubernetes.io/projected/2dd3231a-0da1-44e5-a881-5544caae8e8b-kube-api-access-vb44g\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503667 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-run-k8s-cni-cncf-io\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503625 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-cnibin\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503720 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-system-cni-dir\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.505215 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503674 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6af1e2e-1a39-4433-80a8-563a45ebc689-system-cni-dir\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503762 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-var-lib-cni-bin\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503765 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6af1e2e-1a39-4433-80a8-563a45ebc689-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503789 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hcsk\" (UniqueName: \"kubernetes.io/projected/83ecea52-7a59-4422-9428-94dfaacff497-kube-api-access-2hcsk\") pod \"iptables-alerter-n4x4g\" (UID: \"83ecea52-7a59-4422-9428-94dfaacff497\") " pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503816 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-host-var-lib-cni-bin\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503816 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-registration-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503854 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-modprobe-d\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503880 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-sysconfig\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503905 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-sysctl-d\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503928 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-tmp\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503953 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32792c2d-0873-44aa-9312-9b7d434bd6b8-agent-certs\") pod \"konnectivity-agent-kpqcc\" (UID: \"32792c2d-0873-44aa-9312-9b7d434bd6b8\") " pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503961 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83ecea52-7a59-4422-9428-94dfaacff497-iptables-alerter-script\") pod \"iptables-alerter-n4x4g\" (UID: \"83ecea52-7a59-4422-9428-94dfaacff497\") " pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503978 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-systemd-units\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.503989 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6af1e2e-1a39-4433-80a8-563a45ebc689-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504004 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-run-netns\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504022 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-systemd-units\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504045 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.505837 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504050 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-run-netns\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504076 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504100 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3777315e-9137-4504-b546-c35d39f4613b-tmp-dir\") pod \"node-resolver-99gm8\" (UID: \"3777315e-9137-4504-b546-c35d39f4613b\") " pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504124 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqxq6\" (UniqueName: \"kubernetes.io/projected/d5073278-ac92-4fcd-a3bf-236f10e39dad-kube-api-access-xqxq6\") pod \"node-ca-sx82n\" (UID: \"d5073278-ac92-4fcd-a3bf-236f10e39dad\") " pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504154 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dd3231a-0da1-44e5-a881-5544caae8e8b-ovnkube-script-lib\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504162 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32792c2d-0873-44aa-9312-9b7d434bd6b8-konnectivity-ca\") pod \"konnectivity-agent-kpqcc\" (UID: \"32792c2d-0873-44aa-9312-9b7d434bd6b8\") " pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504188 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91637e1c-200e-429b-ac75-147a1cc1f222-cni-binary-copy\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504206 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhr87\" (UniqueName: \"kubernetes.io/projected/344add4f-e88f-4616-8fb4-8fdbe15da3f7-kube-api-access-dhr87\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504224 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-sysctl-conf\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504245 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504268 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dd3231a-0da1-44e5-a881-5544caae8e8b-ovn-node-metrics-cert\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504294 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-multus-cni-dir\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504310 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dd3231a-0da1-44e5-a881-5544caae8e8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504321 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79bj\" (UniqueName: \"kubernetes.io/projected/91637e1c-200e-429b-ac75-147a1cc1f222-kube-api-access-m79bj\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504360 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-run\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504411 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5073278-ac92-4fcd-a3bf-236f10e39dad-serviceca\") pod \"node-ca-sx82n\" (UID: \"d5073278-ac92-4fcd-a3bf-236f10e39dad\") " pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504655 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91637e1c-200e-429b-ac75-147a1cc1f222-multus-cni-dir\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.506441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504658 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91637e1c-200e-429b-ac75-147a1cc1f222-cni-binary-copy\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.506991 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.504764 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 12:53:06.508057 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.508035 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dd3231a-0da1-44e5-a881-5544caae8e8b-ovn-node-metrics-cert\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.520853 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.520826 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 12:53:06.520853 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.520857 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 12:53:06.521036 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.520870 2554 projected.go:196] Error preparing data for projected volume kube-api-access-mj7sb for pod openshift-network-diagnostics/network-check-target-jwxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:06.521036 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.520939 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb podName:3e72bde8-aa26-4b4a-9929-be0a7b6594d4 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:07.020916755 +0000 UTC m=+3.024218854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mj7sb" (UniqueName: "kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb") pod "network-check-target-jwxch" (UID: "3e72bde8-aa26-4b4a-9929-be0a7b6594d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:06.524846 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.524817 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x42l\" (UniqueName: \"kubernetes.io/projected/b6af1e2e-1a39-4433-80a8-563a45ebc689-kube-api-access-9x42l\") pod \"multus-additional-cni-plugins-6ljtj\" (UID: \"b6af1e2e-1a39-4433-80a8-563a45ebc689\") " pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.530869 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.530836 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb44g\" (UniqueName: \"kubernetes.io/projected/2dd3231a-0da1-44e5-a881-5544caae8e8b-kube-api-access-vb44g\") pod \"ovnkube-node-jtlpz\" (UID: \"2dd3231a-0da1-44e5-a881-5544caae8e8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.531008 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.530992 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hcsk\" (UniqueName: \"kubernetes.io/projected/83ecea52-7a59-4422-9428-94dfaacff497-kube-api-access-2hcsk\") pod \"iptables-alerter-n4x4g\" (UID: \"83ecea52-7a59-4422-9428-94dfaacff497\") " pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.531867 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.531843 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdkn\" (UniqueName: \"kubernetes.io/projected/415e7d0e-e69a-46d7-b32c-ad60c5487fab-kube-api-access-9mdkn\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:06.532753 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.532729 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79bj\" (UniqueName: \"kubernetes.io/projected/91637e1c-200e-429b-ac75-147a1cc1f222-kube-api-access-m79bj\") pod \"multus-vcl2c\" (UID: \"91637e1c-200e-429b-ac75-147a1cc1f222\") " pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.605638 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605609 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-run\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.605638 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605644 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5073278-ac92-4fcd-a3bf-236f10e39dad-serviceca\") pod \"node-ca-sx82n\" (UID: \"d5073278-ac92-4fcd-a3bf-236f10e39dad\") " pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605666 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxxb\" (UniqueName: \"kubernetes.io/projected/3777315e-9137-4504-b546-c35d39f4613b-kube-api-access-7fxxb\") pod \"node-resolver-99gm8\" (UID: \"3777315e-9137-4504-b546-c35d39f4613b\") " pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605689 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5073278-ac92-4fcd-a3bf-236f10e39dad-host\") pod \"node-ca-sx82n\" (UID: \"d5073278-ac92-4fcd-a3bf-236f10e39dad\") " pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605716 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605750 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-sys\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605756 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-run\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605776 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-socket-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605812 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5073278-ac92-4fcd-a3bf-236f10e39dad-host\") pod \"node-ca-sx82n\" (UID: \"d5073278-ac92-4fcd-a3bf-236f10e39dad\") " pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605828 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-device-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605850 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbea5b3b-f397-4df0-a942-9b238c3d15aa-dbus\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.605862 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605872 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-sys\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605874 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-systemd\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.605896 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605903 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-lib-modules\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605916 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-systemd\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:06.605921 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret podName:cbea5b3b-f397-4df0-a942-9b238c3d15aa nodeName:}" failed. No retries permitted until 2026-07-24 12:53:07.105899681 +0000 UTC m=+3.109201776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret") pod "global-pull-secret-syncer-ltm9n" (UID: "cbea5b3b-f397-4df0-a942-9b238c3d15aa") : object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.605982 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbea5b3b-f397-4df0-a942-9b238c3d15aa-dbus\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606014 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-host\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606027 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-lib-modules\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606026 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-socket-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606064 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-device-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606079 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3777315e-9137-4504-b546-c35d39f4613b-hosts-file\") pod \"node-resolver-99gm8\" (UID: \"3777315e-9137-4504-b546-c35d39f4613b\") " pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606104 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjv5\" (UniqueName: \"kubernetes.io/projected/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-kube-api-access-ckjv5\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606121 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3777315e-9137-4504-b546-c35d39f4613b-hosts-file\") pod \"node-resolver-99gm8\" (UID: \"3777315e-9137-4504-b546-c35d39f4613b\") " pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606082 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-host\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606133 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-sys-fs\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606133 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5073278-ac92-4fcd-a3bf-236f10e39dad-serviceca\") pod \"node-ca-sx82n\" (UID: \"d5073278-ac92-4fcd-a3bf-236f10e39dad\") " pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606157 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-kubernetes\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606182 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-sys-fs\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606192 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-kubernetes\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.606396 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606198 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-var-lib-kubelet\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606223 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbea5b3b-f397-4df0-a942-9b238c3d15aa-kubelet-config\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606237 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-tuned\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606255 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-registration-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606274 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-modprobe-d\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606282 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbea5b3b-f397-4df0-a942-9b238c3d15aa-kubelet-config\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606299 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-sysconfig\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606311 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-var-lib-kubelet\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606322 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-sysctl-d\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606325 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-registration-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606352 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-tmp\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606368 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32792c2d-0873-44aa-9312-9b7d434bd6b8-agent-certs\") pod \"konnectivity-agent-kpqcc\" (UID: \"32792c2d-0873-44aa-9312-9b7d434bd6b8\") " pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606377 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-modprobe-d\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606388 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606396 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-sysconfig\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606403 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606441 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3777315e-9137-4504-b546-c35d39f4613b-tmp-dir\") pod \"node-resolver-99gm8\" (UID: \"3777315e-9137-4504-b546-c35d39f4613b\") " pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.607240 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606457 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.608064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606467 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqxq6\" (UniqueName: \"kubernetes.io/projected/d5073278-ac92-4fcd-a3bf-236f10e39dad-kube-api-access-xqxq6\") pod \"node-ca-sx82n\" (UID: \"d5073278-ac92-4fcd-a3bf-236f10e39dad\") " pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.608064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606493 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32792c2d-0873-44aa-9312-9b7d434bd6b8-konnectivity-ca\") pod \"konnectivity-agent-kpqcc\" (UID: \"32792c2d-0873-44aa-9312-9b7d434bd6b8\") " pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:06.608064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606519 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhr87\" (UniqueName: \"kubernetes.io/projected/344add4f-e88f-4616-8fb4-8fdbe15da3f7-kube-api-access-dhr87\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.608064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606544 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-sysctl-conf\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.608064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606705 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-sysctl-conf\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.608064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606748 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3777315e-9137-4504-b546-c35d39f4613b-tmp-dir\") pod \"node-resolver-99gm8\" (UID: \"3777315e-9137-4504-b546-c35d39f4613b\") " pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.608064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606827 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/344add4f-e88f-4616-8fb4-8fdbe15da3f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.608064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.606888 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-sysctl-d\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.608064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.607184 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32792c2d-0873-44aa-9312-9b7d434bd6b8-konnectivity-ca\") pod \"konnectivity-agent-kpqcc\" (UID: \"32792c2d-0873-44aa-9312-9b7d434bd6b8\") " pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:06.608771 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.608750 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-tmp\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.608855 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.608814 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-etc-tuned\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.609053 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.609038 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32792c2d-0873-44aa-9312-9b7d434bd6b8-agent-certs\") pod \"konnectivity-agent-kpqcc\" (UID: \"32792c2d-0873-44aa-9312-9b7d434bd6b8\") " pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:06.617478 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.617456 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxxb\" (UniqueName: \"kubernetes.io/projected/3777315e-9137-4504-b546-c35d39f4613b-kube-api-access-7fxxb\") pod \"node-resolver-99gm8\" (UID: \"3777315e-9137-4504-b546-c35d39f4613b\") " pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.617867 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.617845 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjv5\" (UniqueName: \"kubernetes.io/projected/93bef7ab-aa29-46a3-9e0a-a39466ab6fe3-kube-api-access-ckjv5\") pod \"tuned-zqf8d\" (UID: \"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3\") " pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.618364 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.618087 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqxq6\" (UniqueName: \"kubernetes.io/projected/d5073278-ac92-4fcd-a3bf-236f10e39dad-kube-api-access-xqxq6\") pod \"node-ca-sx82n\" (UID: \"d5073278-ac92-4fcd-a3bf-236f10e39dad\") " pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.618901 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.618870 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhr87\" (UniqueName: \"kubernetes.io/projected/344add4f-e88f-4616-8fb4-8fdbe15da3f7-kube-api-access-dhr87\") pod \"aws-ebs-csi-driver-node-rdjwl\" (UID: \"344add4f-e88f-4616-8fb4-8fdbe15da3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.695618 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.695527 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:06.705878 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.705858 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" Jul 24 12:53:06.715072 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.715049 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vcl2c" Jul 24 12:53:06.720743 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.720725 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n4x4g" Jul 24 12:53:06.729302 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.729283 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" Jul 24 12:53:06.742129 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.742112 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-99gm8" Jul 24 12:53:06.750861 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.750845 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" Jul 24 12:53:06.756348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.756325 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sx82n" Jul 24 12:53:06.760914 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:06.760895 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:07.008933 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.008849 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:07.009063 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.008974 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:07.009063 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.009029 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs podName:415e7d0e-e69a-46d7-b32c-ad60c5487fab nodeName:}" failed. No retries permitted until 2026-07-24 12:53:08.009011952 +0000 UTC m=+4.012314036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs") pod "network-metrics-daemon-jcst2" (UID: "415e7d0e-e69a-46d7-b32c-ad60c5487fab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:07.109144 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.109116 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:07.109294 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.109157 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:07.109294 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.109271 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:07.109294 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.109293 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 12:53:07.109412 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.109307 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 12:53:07.109412 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.109320 2554 projected.go:196] Error preparing data for projected volume kube-api-access-mj7sb for pod openshift-network-diagnostics/network-check-target-jwxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:07.109412 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.109337 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret podName:cbea5b3b-f397-4df0-a942-9b238c3d15aa nodeName:}" failed. No retries permitted until 2026-07-24 12:53:08.109318508 +0000 UTC m=+4.112620607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret") pod "global-pull-secret-syncer-ltm9n" (UID: "cbea5b3b-f397-4df0-a942-9b238c3d15aa") : object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:07.109412 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.109353 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb podName:3e72bde8-aa26-4b4a-9929-be0a7b6594d4 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:08.109343885 +0000 UTC m=+4.112645980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj7sb" (UniqueName: "kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb") pod "network-check-target-jwxch" (UID: "3e72bde8-aa26-4b4a-9929-be0a7b6594d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:07.440017 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:07.439984 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3777315e_9137_4504_b546_c35d39f4613b.slice/crio-f69108338abcbd982bd74ec85a296dc1f4b3209bd17c349c953938e863201e29 WatchSource:0}: Error finding container f69108338abcbd982bd74ec85a296dc1f4b3209bd17c349c953938e863201e29: Status 404 returned error can't find the container with id f69108338abcbd982bd74ec85a296dc1f4b3209bd17c349c953938e863201e29 Jul 24 12:53:07.440983 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:07.440946 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ecea52_7a59_4422_9428_94dfaacff497.slice/crio-403714e69740f804618c173fc233a83de742180bc3e8cfb72697ba72a9696cf8 WatchSource:0}: Error finding container 403714e69740f804618c173fc233a83de742180bc3e8cfb72697ba72a9696cf8: Status 404 returned error can't find the container with id 403714e69740f804618c173fc233a83de742180bc3e8cfb72697ba72a9696cf8 Jul 24 12:53:07.442496 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:07.442436 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344add4f_e88f_4616_8fb4_8fdbe15da3f7.slice/crio-6c164628ae8bf425dc813a600b29def99223c2313741df7877fdf8034cd357a6 WatchSource:0}: Error finding container 6c164628ae8bf425dc813a600b29def99223c2313741df7877fdf8034cd357a6: Status 404 returned error can't find the container with id 6c164628ae8bf425dc813a600b29def99223c2313741df7877fdf8034cd357a6 Jul 24 12:53:07.446206 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:07.445949 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd3231a_0da1_44e5_a881_5544caae8e8b.slice/crio-abb8fa6eec327992e65f877b83ae9d3317b027f5d9016828e051ab5f8b4820f0 WatchSource:0}: Error finding container abb8fa6eec327992e65f877b83ae9d3317b027f5d9016828e051ab5f8b4820f0: Status 404 returned error can't find the container with id abb8fa6eec327992e65f877b83ae9d3317b027f5d9016828e051ab5f8b4820f0 Jul 24 12:53:07.447883 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:07.447856 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91637e1c_200e_429b_ac75_147a1cc1f222.slice/crio-742ff3a7805a396f051a89a0b5253c2f6da8cb035fc13fc35bd43455b754263c WatchSource:0}: Error finding container 742ff3a7805a396f051a89a0b5253c2f6da8cb035fc13fc35bd43455b754263c: Status 404 returned error can't find the container with id 742ff3a7805a396f051a89a0b5253c2f6da8cb035fc13fc35bd43455b754263c Jul 24 12:53:07.449415 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:07.449391 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bef7ab_aa29_46a3_9e0a_a39466ab6fe3.slice/crio-b73329fb0964220bc222c581d530964c68359dee3c43d7b3b41149479fd2de31 WatchSource:0}: Error finding container b73329fb0964220bc222c581d530964c68359dee3c43d7b3b41149479fd2de31: Status 404 returned error can't find the container with id b73329fb0964220bc222c581d530964c68359dee3c43d7b3b41149479fd2de31 Jul 24 12:53:07.455719 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.455687 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 12:48:06 +0000 UTC" deadline="2028-03-07 23:14:58.40206478 +0000 UTC" Jul 24 12:53:07.455792 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.455719 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14218h21m50.946349521s" Jul 24 12:53:07.457324 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:07.456871 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6af1e2e_1a39_4433_80a8_563a45ebc689.slice/crio-a050f67834a05fdec4b6e0406d7864a2569a06deb00f85f72a13a3f0d069cbc9 WatchSource:0}: Error finding container a050f67834a05fdec4b6e0406d7864a2569a06deb00f85f72a13a3f0d069cbc9: Status 404 returned error can't find the container with id a050f67834a05fdec4b6e0406d7864a2569a06deb00f85f72a13a3f0d069cbc9 Jul 24 12:53:07.593814 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.593628 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:07.593973 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.593679 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:07.593973 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.593893 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:07.594070 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:07.593973 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:07.601096 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.601073 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" event={"ID":"b6af1e2e-1a39-4433-80a8-563a45ebc689","Type":"ContainerStarted","Data":"a050f67834a05fdec4b6e0406d7864a2569a06deb00f85f72a13a3f0d069cbc9"} Jul 24 12:53:07.604847 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.603294 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kpqcc" event={"ID":"32792c2d-0873-44aa-9312-9b7d434bd6b8","Type":"ContainerStarted","Data":"4735d513b2a9373530fcefce0c577c21ae1c7552a0c23cd205d827934c797ef4"} Jul 24 12:53:07.608533 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.605683 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" event={"ID":"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3","Type":"ContainerStarted","Data":"b73329fb0964220bc222c581d530964c68359dee3c43d7b3b41149479fd2de31"} Jul 24 12:53:07.609589 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.609547 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" event={"ID":"2dd3231a-0da1-44e5-a881-5544caae8e8b","Type":"ContainerStarted","Data":"abb8fa6eec327992e65f877b83ae9d3317b027f5d9016828e051ab5f8b4820f0"} Jul 24 12:53:07.610910 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.610833 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-99gm8" event={"ID":"3777315e-9137-4504-b546-c35d39f4613b","Type":"ContainerStarted","Data":"f69108338abcbd982bd74ec85a296dc1f4b3209bd17c349c953938e863201e29"} Jul 24 12:53:07.612772 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.612752 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal" event={"ID":"7de245839c2e01ba31555968cc0536f8","Type":"ContainerStarted","Data":"e081e7eb5479306e91d32f39b6c6b8af390686fe72ee7ae020e3e65e61b9b393"} Jul 24 12:53:07.613806 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.613787 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sx82n" event={"ID":"d5073278-ac92-4fcd-a3bf-236f10e39dad","Type":"ContainerStarted","Data":"00961fd98a55a26dc38f75c352e0202bbf05599e02985e3c6cd79a736f4df08f"} Jul 24 12:53:07.614815 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.614798 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vcl2c" event={"ID":"91637e1c-200e-429b-ac75-147a1cc1f222","Type":"ContainerStarted","Data":"742ff3a7805a396f051a89a0b5253c2f6da8cb035fc13fc35bd43455b754263c"} Jul 24 12:53:07.615859 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.615827 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n4x4g" event={"ID":"83ecea52-7a59-4422-9428-94dfaacff497","Type":"ContainerStarted","Data":"403714e69740f804618c173fc233a83de742180bc3e8cfb72697ba72a9696cf8"} Jul 24 12:53:07.616880 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.616858 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" event={"ID":"344add4f-e88f-4616-8fb4-8fdbe15da3f7","Type":"ContainerStarted","Data":"6c164628ae8bf425dc813a600b29def99223c2313741df7877fdf8034cd357a6"} Jul 24 12:53:07.630505 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:07.630470 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-100.ec2.internal" podStartSLOduration=2.630458095 podStartE2EDuration="2.630458095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:07.63005923 +0000 UTC m=+3.633361334" watchObservedRunningTime="2026-07-24 12:53:07.630458095 +0000 UTC m=+3.633760200" Jul 24 12:53:08.017217 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:08.016643 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:08.017217 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:08.016824 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:08.017217 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:08.016892 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs podName:415e7d0e-e69a-46d7-b32c-ad60c5487fab nodeName:}" failed. No retries permitted until 2026-07-24 12:53:10.01687303 +0000 UTC m=+6.020175120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs") pod "network-metrics-daemon-jcst2" (UID: "415e7d0e-e69a-46d7-b32c-ad60c5487fab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:08.117549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:08.117511 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:08.117750 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:08.117588 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:08.117820 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:08.117777 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 12:53:08.117820 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:08.117798 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 12:53:08.117820 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:08.117812 2554 projected.go:196] Error preparing data for projected volume kube-api-access-mj7sb for pod openshift-network-diagnostics/network-check-target-jwxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:08.117962 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:08.117873 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb podName:3e72bde8-aa26-4b4a-9929-be0a7b6594d4 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:10.117854012 +0000 UTC m=+6.121156109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj7sb" (UniqueName: "kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb") pod "network-check-target-jwxch" (UID: "3e72bde8-aa26-4b4a-9929-be0a7b6594d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:08.117962 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:08.117946 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:08.118074 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:08.117983 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret podName:cbea5b3b-f397-4df0-a942-9b238c3d15aa nodeName:}" failed. No retries permitted until 2026-07-24 12:53:10.117971632 +0000 UTC m=+6.121273727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret") pod "global-pull-secret-syncer-ltm9n" (UID: "cbea5b3b-f397-4df0-a942-9b238c3d15aa") : object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:08.456331 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:08.456263 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 12:48:06 +0000 UTC" deadline="2028-01-18 04:18:49.019073238 +0000 UTC" Jul 24 12:53:08.456331 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:08.456331 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13023h25m40.562747489s" Jul 24 12:53:08.595820 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:08.595318 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:08.595820 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:08.595445 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:08.623740 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:08.623704 2554 generic.go:356] "Generic (PLEG): container finished" podID="aeed6dc75c412f2f51097cace09f376d" containerID="1202c90f160df1a9c516db6408faca3a91e9f6c63ec8834604799d1dbe501ba1" exitCode=0 Jul 24 12:53:08.624627 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:08.624601 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" event={"ID":"aeed6dc75c412f2f51097cace09f376d","Type":"ContainerDied","Data":"1202c90f160df1a9c516db6408faca3a91e9f6c63ec8834604799d1dbe501ba1"} Jul 24 12:53:09.592742 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:09.592709 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:09.593221 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:09.592848 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:09.593221 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:09.592897 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:09.593221 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:09.593017 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:09.640209 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:09.640179 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" event={"ID":"aeed6dc75c412f2f51097cace09f376d","Type":"ContainerStarted","Data":"d5297b4b8316ca85a9359feef24e1b4cb1efa49669b12523a189a05d64cdb4f0"} Jul 24 12:53:10.035227 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:10.035139 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:10.035398 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:10.035332 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:10.035542 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:10.035400 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs podName:415e7d0e-e69a-46d7-b32c-ad60c5487fab nodeName:}" failed. No retries permitted until 2026-07-24 12:53:14.035381968 +0000 UTC m=+10.038684056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs") pod "network-metrics-daemon-jcst2" (UID: "415e7d0e-e69a-46d7-b32c-ad60c5487fab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:10.136592 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:10.135742 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:10.136592 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:10.135802 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:10.136592 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:10.135988 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 12:53:10.136592 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:10.136009 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 12:53:10.136592 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:10.136022 2554 projected.go:196] Error preparing data for projected volume kube-api-access-mj7sb for pod openshift-network-diagnostics/network-check-target-jwxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:10.136592 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:10.136085 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb podName:3e72bde8-aa26-4b4a-9929-be0a7b6594d4 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:14.136067067 +0000 UTC m=+10.139369168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj7sb" (UniqueName: "kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb") pod "network-check-target-jwxch" (UID: "3e72bde8-aa26-4b4a-9929-be0a7b6594d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:10.136592 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:10.136489 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:10.136592 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:10.136539 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret podName:cbea5b3b-f397-4df0-a942-9b238c3d15aa nodeName:}" failed. No retries permitted until 2026-07-24 12:53:14.13652484 +0000 UTC m=+10.139826938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret") pod "global-pull-secret-syncer-ltm9n" (UID: "cbea5b3b-f397-4df0-a942-9b238c3d15aa") : object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:10.593203 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:10.592679 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:10.593203 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:10.592835 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:11.593818 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:11.593255 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:11.593818 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:11.593414 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:11.593818 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:11.593448 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:11.593818 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:11.593578 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:12.592874 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:12.592799 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:12.593017 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:12.592921 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:13.592952 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:13.592916 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:13.593413 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:13.592970 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:13.593413 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:13.593063 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:13.593413 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:13.593191 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:14.067028 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:14.066945 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:14.067181 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:14.067064 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:14.067181 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:14.067115 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs podName:415e7d0e-e69a-46d7-b32c-ad60c5487fab nodeName:}" failed. No retries permitted until 2026-07-24 12:53:22.067101115 +0000 UTC m=+18.070403196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs") pod "network-metrics-daemon-jcst2" (UID: "415e7d0e-e69a-46d7-b32c-ad60c5487fab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:14.168030 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:14.167996 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:14.168030 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:14.168039 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:14.168265 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:14.168156 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:14.168265 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:14.168170 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 12:53:14.168265 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:14.168183 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 12:53:14.168265 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:14.168192 2554 projected.go:196] Error preparing data for projected volume kube-api-access-mj7sb for pod openshift-network-diagnostics/network-check-target-jwxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:14.168265 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:14.168222 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret podName:cbea5b3b-f397-4df0-a942-9b238c3d15aa nodeName:}" failed. No retries permitted until 2026-07-24 12:53:22.16820777 +0000 UTC m=+18.171509855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret") pod "global-pull-secret-syncer-ltm9n" (UID: "cbea5b3b-f397-4df0-a942-9b238c3d15aa") : object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:14.168265 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:14.168236 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb podName:3e72bde8-aa26-4b4a-9929-be0a7b6594d4 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:22.16823047 +0000 UTC m=+18.171532552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj7sb" (UniqueName: "kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb") pod "network-check-target-jwxch" (UID: "3e72bde8-aa26-4b4a-9929-be0a7b6594d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:14.595473 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:14.595433 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:14.595987 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:14.595547 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:15.593348 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:15.593310 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:15.593517 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:15.593466 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:15.593906 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:15.593883 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:15.594010 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:15.593993 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:16.593587 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:16.593501 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:16.594062 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:16.593639 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:17.593195 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:17.593158 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:17.593378 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:17.593300 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:17.593550 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:17.593526 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:17.593705 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:17.593681 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:18.593196 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:18.593114 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:18.593354 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:18.593260 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:19.593418 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:19.593382 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:19.593814 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:19.593497 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:19.593814 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:19.593531 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:19.593814 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:19.593628 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:20.593386 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:20.593346 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:20.593548 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:20.593476 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:21.593156 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:21.593115 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:21.593395 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:21.593172 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:21.593395 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:21.593261 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:21.593500 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:21.593405 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:22.127308 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:22.127265 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:22.127868 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:22.127415 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:22.127868 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:22.127490 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs podName:415e7d0e-e69a-46d7-b32c-ad60c5487fab nodeName:}" failed. No retries permitted until 2026-07-24 12:53:38.127471875 +0000 UTC m=+34.130773971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs") pod "network-metrics-daemon-jcst2" (UID: "415e7d0e-e69a-46d7-b32c-ad60c5487fab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:22.227751 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:22.227715 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:22.227961 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:22.227773 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:22.227961 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:22.227901 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:22.227961 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:22.227911 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 12:53:22.227961 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:22.227934 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 12:53:22.227961 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:22.227947 2554 projected.go:196] Error preparing data for projected volume kube-api-access-mj7sb for pod openshift-network-diagnostics/network-check-target-jwxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:22.228179 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:22.227971 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret podName:cbea5b3b-f397-4df0-a942-9b238c3d15aa nodeName:}" failed. No retries permitted until 2026-07-24 12:53:38.227954264 +0000 UTC m=+34.231256349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret") pod "global-pull-secret-syncer-ltm9n" (UID: "cbea5b3b-f397-4df0-a942-9b238c3d15aa") : object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:22.228179 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:22.227993 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb podName:3e72bde8-aa26-4b4a-9929-be0a7b6594d4 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:38.227979225 +0000 UTC m=+34.231281325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj7sb" (UniqueName: "kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb") pod "network-check-target-jwxch" (UID: "3e72bde8-aa26-4b4a-9929-be0a7b6594d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:22.593194 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:22.593158 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:22.593379 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:22.593278 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:23.593301 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:23.593267 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:23.593778 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:23.593275 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:23.593778 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:23.593374 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:23.593778 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:23.593493 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:24.593652 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:24.593614 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:24.594134 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:24.593703 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:25.592974 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:25.592934 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:25.592974 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:25.592971 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:25.593196 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:25.593075 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:25.593196 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:25.593183 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:26.596443 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.596289 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:26.597116 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:26.596519 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:26.674975 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.674798 2554 generic.go:356] "Generic (PLEG): container finished" podID="b6af1e2e-1a39-4433-80a8-563a45ebc689" containerID="a2dfec35e2c2ef52983b7bdb826c44d74466ecb62e089b1f606fd9d8f39b9383" exitCode=0 Jul 24 12:53:26.675104 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.674873 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" event={"ID":"b6af1e2e-1a39-4433-80a8-563a45ebc689","Type":"ContainerDied","Data":"a2dfec35e2c2ef52983b7bdb826c44d74466ecb62e089b1f606fd9d8f39b9383"} Jul 24 12:53:26.676676 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.676647 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kpqcc" event={"ID":"32792c2d-0873-44aa-9312-9b7d434bd6b8","Type":"ContainerStarted","Data":"c0b2ff6f528314d6233268b590f87b7461c3bb93ae7031372244d31ffdd17069"} Jul 24 12:53:26.678212 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.678185 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" event={"ID":"93bef7ab-aa29-46a3-9e0a-a39466ab6fe3","Type":"ContainerStarted","Data":"b128ab880917d8e6ab6edaa167da43b469899d2dd0db8b24b8b4be828c1ad504"} Jul 24 12:53:26.680080 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.680057 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" event={"ID":"2dd3231a-0da1-44e5-a881-5544caae8e8b","Type":"ContainerStarted","Data":"90be5a446d029fbbfcd3a148f99b7f8d2771235d75f3c57588588fe744a53c3a"} Jul 24 12:53:26.680184 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.680135 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" event={"ID":"2dd3231a-0da1-44e5-a881-5544caae8e8b","Type":"ContainerStarted","Data":"b5e14c0891d36a5144b875cb3ab2b70281450d34a83f0434eb68ace61a94b5d4"} Jul 24 12:53:26.681442 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.681423 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-99gm8" event={"ID":"3777315e-9137-4504-b546-c35d39f4613b","Type":"ContainerStarted","Data":"1cb2dbe86f1e02d3ddd26c356f018e5935737598cf58633e7b230a9ce0e5829d"} Jul 24 12:53:26.682762 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.682729 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sx82n" event={"ID":"d5073278-ac92-4fcd-a3bf-236f10e39dad","Type":"ContainerStarted","Data":"908ead39a632600a4c6e7dadaa4cbbcd81a5b7b09ca306b27a2ace055e6e2beb"} Jul 24 12:53:26.684321 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.684300 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vcl2c" event={"ID":"91637e1c-200e-429b-ac75-147a1cc1f222","Type":"ContainerStarted","Data":"fcd46c527a708a711aedcabb7547d81a1378ecdd490493a3ca2deab0a9083add"} Jul 24 12:53:26.685786 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.685758 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" event={"ID":"344add4f-e88f-4616-8fb4-8fdbe15da3f7","Type":"ContainerStarted","Data":"1e1d080f9001568f2d5d0974c730aa112b0c3f6391fd8afb60cc6491d6fdbc86"} Jul 24 12:53:26.699242 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.699199 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-100.ec2.internal" podStartSLOduration=21.699185087 podStartE2EDuration="21.699185087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:09.658841429 +0000 UTC m=+5.662143534" watchObservedRunningTime="2026-07-24 12:53:26.699185087 +0000 UTC m=+22.702487192" Jul 24 12:53:26.716207 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.716154 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-99gm8" podStartSLOduration=3.899805443 podStartE2EDuration="22.716133794s" totalImagesPullingTime="18.816328351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:26.715307206 +0000 UTC m=+22.718609313" watchObservedRunningTime="2026-07-24 12:53:26.716133794 +0000 UTC m=+22.719435914" Jul 24 12:53:26.734359 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.734321 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zqf8d" podStartSLOduration=3.911410971 podStartE2EDuration="22.734304635s" totalImagesPullingTime="18.822893664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:26.733731821 +0000 UTC m=+22.737033924" watchObservedRunningTime="2026-07-24 12:53:26.734304635 +0000 UTC m=+22.737606738" Jul 24 12:53:26.748803 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.748761 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sx82n" podStartSLOduration=3.968210957 podStartE2EDuration="22.748746694s" totalImagesPullingTime="18.780535737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:26.748414832 +0000 UTC m=+22.751716937" watchObservedRunningTime="2026-07-24 12:53:26.748746694 +0000 UTC m=+22.752048827" Jul 24 12:53:26.767282 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.767205 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kpqcc" podStartSLOduration=4.37483993 podStartE2EDuration="22.767186538s" totalImagesPullingTime="18.392346608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:26.766398256 +0000 UTC m=+22.769700363" watchObservedRunningTime="2026-07-24 12:53:26.767186538 +0000 UTC m=+22.770488643" Jul 24 12:53:26.787428 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.787382 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vcl2c" podStartSLOduration=3.9273726140000003 podStartE2EDuration="22.787365794s" totalImagesPullingTime="18.85999318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:26.78714193 +0000 UTC m=+22.790444054" watchObservedRunningTime="2026-07-24 12:53:26.787365794 +0000 UTC m=+22.790667897" Jul 24 12:53:26.849879 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.849845 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:26.915940 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:26.915873 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:27.016744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:27.016702 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:27.593322 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:27.593299 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:27.593454 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:27.593321 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:27.593454 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:27.593404 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:27.593526 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:27.593479 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:27.692362 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:27.691818 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" event={"ID":"2dd3231a-0da1-44e5-a881-5544caae8e8b","Type":"ContainerStarted","Data":"174bcf62e03226fe9378609dd5cd0edf2bad96311290fefcf1444ebb1a88f23e"} Jul 24 12:53:27.692362 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:27.691901 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" event={"ID":"2dd3231a-0da1-44e5-a881-5544caae8e8b","Type":"ContainerStarted","Data":"a8d60acb6c0a42e7e9889236cd96fe12098988af4947c0bb0d54e569f0efd9a3"} Jul 24 12:53:27.692362 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:27.691956 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" event={"ID":"2dd3231a-0da1-44e5-a881-5544caae8e8b","Type":"ContainerStarted","Data":"f6a5c4c7769c4afc9ebaaac5b134f9d02347be049c188e28038068d8be16d5f1"} Jul 24 12:53:27.692362 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:27.692005 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" event={"ID":"2dd3231a-0da1-44e5-a881-5544caae8e8b","Type":"ContainerStarted","Data":"4e31d729e406a03f259b0bbb9baef76b4611a11e2043e4846dfd3a4f48a9965d"} Jul 24 12:53:27.724071 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:27.723478 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 12:53:28.460773 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:28.460651 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T12:53:27.723495886Z","UUID":"2d8ba9f2-aca7-4431-8e47-85c7aa85498f","Handler":null,"Name":"","Endpoint":""} Jul 24 12:53:28.462336 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:28.462314 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 12:53:28.462514 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:28.462358 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 12:53:28.593795 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:28.593755 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:28.593982 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:28.593910 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:28.696081 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:28.696043 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n4x4g" event={"ID":"83ecea52-7a59-4422-9428-94dfaacff497","Type":"ContainerStarted","Data":"0a39b35e586e0f44f2fc606bc5b8ffa88dd374e81f6534f74b5d9444963edb10"} Jul 24 12:53:28.701077 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:28.701043 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" event={"ID":"344add4f-e88f-4616-8fb4-8fdbe15da3f7","Type":"ContainerStarted","Data":"ef96b81db62094d89788a3d72239eeea2cf5365e5c15a87a55ab9c5d61a0d90f"} Jul 24 12:53:28.714250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:28.714170 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n4x4g" podStartSLOduration=5.878266645 podStartE2EDuration="24.71415675s" totalImagesPullingTime="18.835890105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:28.713840546 +0000 UTC m=+24.717142651" watchObservedRunningTime="2026-07-24 12:53:28.71415675 +0000 UTC m=+24.717458864" Jul 24 12:53:28.842771 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:28.842738 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:28.942727 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:28.942692 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:29.592861 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:29.592822 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:29.593017 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:29.592864 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:29.593017 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:29.592959 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:29.593122 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:29.593088 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:29.705675 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:29.705641 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" event={"ID":"344add4f-e88f-4616-8fb4-8fdbe15da3f7","Type":"ContainerStarted","Data":"2211a3943ddc2cb84e94cd621e191d3007554d1a1a7c1ce9a5ddb9dcf02b0700"} Jul 24 12:53:29.727132 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:29.727089 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdjwl" podStartSLOduration=4.5432890740000005 podStartE2EDuration="25.727072056s" totalImagesPullingTime="21.183782982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:29.726756271 +0000 UTC m=+25.730058376" watchObservedRunningTime="2026-07-24 12:53:29.727072056 +0000 UTC m=+25.730374160" Jul 24 12:53:30.412195 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:30.412161 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:30.412834 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:30.412814 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:30.596825 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:30.596630 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:30.596986 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:30.596908 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:30.707533 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:30.707457 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:30.707950 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:30.707933 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kpqcc" Jul 24 12:53:30.792464 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:30.792433 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:31.194282 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:31.194257 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:31.292980 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:31.292946 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:31.359018 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:31.358983 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:31.593433 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:31.593407 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:31.593616 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:31.593408 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:31.593616 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:31.593496 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:31.593697 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:31.593647 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:31.711543 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:31.711508 2554 generic.go:356] "Generic (PLEG): container finished" podID="b6af1e2e-1a39-4433-80a8-563a45ebc689" containerID="19852c83998a89c3bd497b232296cc8f4948e2874da673d6848a31f6a9be54a9" exitCode=0 Jul 24 12:53:31.712254 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:31.711603 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" event={"ID":"b6af1e2e-1a39-4433-80a8-563a45ebc689","Type":"ContainerDied","Data":"19852c83998a89c3bd497b232296cc8f4948e2874da673d6848a31f6a9be54a9"} Jul 24 12:53:31.715361 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:31.715329 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" event={"ID":"2dd3231a-0da1-44e5-a881-5544caae8e8b","Type":"ContainerStarted","Data":"3ff9b92ebe0dabb7e8c065a4c557e57b6ec1effec33d44898b0656b5e93e0af7"} Jul 24 12:53:32.595574 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:32.595532 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:32.595721 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:32.595645 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:33.593310 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:33.593107 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:33.593899 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:33.593110 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:33.593899 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:33.593386 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:33.593899 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:33.593506 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:33.721402 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:33.721368 2554 generic.go:356] "Generic (PLEG): container finished" podID="b6af1e2e-1a39-4433-80a8-563a45ebc689" containerID="a8e1936556e96a46cd88c4926ed5f0b925b1629541fd555f442e664c05b1afdc" exitCode=0 Jul 24 12:53:33.721556 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:33.721452 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" event={"ID":"b6af1e2e-1a39-4433-80a8-563a45ebc689","Type":"ContainerDied","Data":"a8e1936556e96a46cd88c4926ed5f0b925b1629541fd555f442e664c05b1afdc"} Jul 24 12:53:33.727466 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:33.727441 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" event={"ID":"2dd3231a-0da1-44e5-a881-5544caae8e8b","Type":"ContainerStarted","Data":"27645f6eeb5f93001995a35e8760a8e16c6046a3724b17f91291ec4a3048bd22"} Jul 24 12:53:33.727805 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:33.727778 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:33.743646 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:33.743625 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:33.775741 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:33.775699 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" podStartSLOduration=10.76260255 podStartE2EDuration="29.775684587s" totalImagesPullingTime="19.013082037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:33.775397545 +0000 UTC m=+29.778699658" watchObservedRunningTime="2026-07-24 12:53:33.775684587 +0000 UTC m=+29.778986690" Jul 24 12:53:34.593708 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:34.593678 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:34.594292 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:34.593783 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:34.732938 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:34.732914 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:34.733091 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:34.733052 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:34.753822 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:34.753798 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:53:34.833700 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:34.833662 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:35.020198 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.019942 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jcst2"] Jul 24 12:53:35.020372 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.020334 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:35.020502 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:35.020472 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:35.020727 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.020701 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ltm9n"] Jul 24 12:53:35.020832 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.020817 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:35.020938 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:35.020917 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:35.021146 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.021124 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jwxch"] Jul 24 12:53:35.021249 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.021207 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:35.021356 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:35.021334 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:35.029098 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.029066 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:35.110509 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.110439 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:35.742013 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.741973 2554 generic.go:356] "Generic (PLEG): container finished" podID="b6af1e2e-1a39-4433-80a8-563a45ebc689" containerID="989504b3b56bf4e92f3ccd2505dd83898ab0e21ab1d92b517cae8c0c87f590f5" exitCode=0 Jul 24 12:53:35.742588 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:35.742516 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" event={"ID":"b6af1e2e-1a39-4433-80a8-563a45ebc689","Type":"ContainerDied","Data":"989504b3b56bf4e92f3ccd2505dd83898ab0e21ab1d92b517cae8c0c87f590f5"} Jul 24 12:53:36.219875 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:36.219843 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:36.316706 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:36.316673 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:36.412178 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:36.412150 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:36.514806 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:36.514721 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:36.593232 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:36.593198 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:36.593401 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:36.593198 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:36.593401 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:36.593337 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:36.593401 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:36.593392 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:36.593650 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:36.593472 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:36.593650 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:36.593544 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:37.120889 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:37.120850 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:37.217973 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:37.217937 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:37.569078 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:37.569049 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-99gm8_3777315e-9137-4504-b546-c35d39f4613b/dns-node-resolver/0.log" Jul 24 12:53:37.826468 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:37.826348 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:37.945392 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:37.945350 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:38.127897 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.127811 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:38.149851 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.149820 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:38.150016 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.149980 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:38.150088 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.150053 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs podName:415e7d0e-e69a-46d7-b32c-ad60c5487fab nodeName:}" failed. No retries permitted until 2026-07-24 12:54:10.150032117 +0000 UTC m=+66.153334198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs") pod "network-metrics-daemon-jcst2" (UID: "415e7d0e-e69a-46d7-b32c-ad60c5487fab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 12:53:38.228652 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.228619 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:38.250656 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.250626 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:38.250656 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.250662 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:38.250892 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.250787 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:38.250892 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.250856 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret podName:cbea5b3b-f397-4df0-a942-9b238c3d15aa nodeName:}" failed. No retries permitted until 2026-07-24 12:54:10.250837081 +0000 UTC m=+66.254139167 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret") pod "global-pull-secret-syncer-ltm9n" (UID: "cbea5b3b-f397-4df0-a942-9b238c3d15aa") : object "kube-system"/"original-pull-secret" not registered Jul 24 12:53:38.250994 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.250791 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 12:53:38.250994 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.250912 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 12:53:38.250994 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.250921 2554 projected.go:196] Error preparing data for projected volume kube-api-access-mj7sb for pod openshift-network-diagnostics/network-check-target-jwxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:38.250994 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.250966 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb podName:3e72bde8-aa26-4b4a-9929-be0a7b6594d4 nodeName:}" failed. No retries permitted until 2026-07-24 12:54:10.250953797 +0000 UTC m=+66.254255879 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj7sb" (UniqueName: "kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb") pod "network-check-target-jwxch" (UID: "3e72bde8-aa26-4b4a-9929-be0a7b6594d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 12:53:38.568790 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.568759 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sx82n_d5073278-ac92-4fcd-a3bf-236f10e39dad/node-ca/0.log" Jul 24 12:53:38.593236 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.593204 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:38.593236 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.593233 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:38.593455 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.593205 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:38.593455 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.593334 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ltm9n" podUID="cbea5b3b-f397-4df0-a942-9b238c3d15aa" Jul 24 12:53:38.593455 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.593430 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcst2" podUID="415e7d0e-e69a-46d7-b32c-ad60c5487fab" Jul 24 12:53:38.593583 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:38.593518 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jwxch" podUID="3e72bde8-aa26-4b4a-9929-be0a7b6594d4" Jul 24 12:53:38.909260 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.909214 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:38.976193 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:38.976160 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:39.309055 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.309026 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-100.ec2.internal" event="NodeReady" Jul 24 12:53:39.309641 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.309176 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 12:53:39.368674 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.368645 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hgx2s"] Jul 24 12:53:39.371230 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.371205 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.372356 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.371653 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8pq82"] Jul 24 12:53:39.373321 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.373306 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8pq82" Jul 24 12:53:39.376961 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.376936 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxgfg\"" Jul 24 12:53:39.377472 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.377456 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 12:53:39.377676 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.377659 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 12:53:39.379938 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.377954 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 12:53:39.379938 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.378591 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 12:53:39.379938 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.379047 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 12:53:39.380623 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.380591 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ttjr9\"" Jul 24 12:53:39.380976 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.380950 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 12:53:39.383763 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.383736 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 12:53:39.385861 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.385122 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgx2s"] Jul 24 12:53:39.393463 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.393442 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8pq82"] Jul 24 12:53:39.400529 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.400511 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mw26d"] Jul 24 12:53:39.402493 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.402474 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.405376 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.405356 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7xh6d\"" Jul 24 12:53:39.405749 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.405731 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 12:53:39.405919 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.405893 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 12:53:39.416161 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.416142 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mw26d"] Jul 24 12:53:39.455330 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.455300 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-jxxl9"] Jul 24 12:53:39.457351 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.457333 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.460815 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.460479 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 12:53:39.460815 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.460649 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 12:53:39.460815 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.460676 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-58c6d\"" Jul 24 12:53:39.460815 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.460729 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 12:53:39.460815 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.460735 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 12:53:39.461550 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.461528 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb1b5900-c451-4a39-b80e-64ee6581fb10-cert\") pod \"ingress-canary-8pq82\" (UID: \"fb1b5900-c451-4a39-b80e-64ee6581fb10\") " pod="openshift-ingress-canary/ingress-canary-8pq82" Jul 24 12:53:39.461650 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.461579 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.461650 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.461608 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knw4\" (UniqueName: \"kubernetes.io/projected/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-api-access-2knw4\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.461650 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.461642 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-data-volume\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.461784 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.461684 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-crio-socket\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.461784 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.461725 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.461784 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.461776 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwkmc\" (UniqueName: \"kubernetes.io/projected/fb1b5900-c451-4a39-b80e-64ee6581fb10-kube-api-access-xwkmc\") pod \"ingress-canary-8pq82\" (UID: \"fb1b5900-c451-4a39-b80e-64ee6581fb10\") " pod="openshift-ingress-canary/ingress-canary-8pq82" Jul 24 12:53:39.466986 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.466967 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-jxxl9"] Jul 24 12:53:39.501209 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.501177 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:39.562391 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562298 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb1b5900-c451-4a39-b80e-64ee6581fb10-cert\") pod \"ingress-canary-8pq82\" (UID: \"fb1b5900-c451-4a39-b80e-64ee6581fb10\") " pod="openshift-ingress-canary/ingress-canary-8pq82" Jul 24 12:53:39.562391 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562340 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.562391 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562370 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knw4\" (UniqueName: \"kubernetes.io/projected/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-api-access-2knw4\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.562705 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562404 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-data-volume\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.562705 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:39.562437 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 12:53:39.562705 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562450 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-crio-socket\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.562705 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562474 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e1b35c-102a-43c8-b556-b4ba82ac1796-tmp-dir\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.562705 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:39.562516 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-rbac-proxy-cm podName:e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:40.062500686 +0000 UTC m=+36.065802803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgx2s" (UID: "e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8") : configmap "kube-rbac-proxy" not found Jul 24 12:53:39.562705 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562541 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/62163913-600f-4a32-adc7-8604e41e1d79-signing-cabundle\") pod \"service-ca-658d7b7c78-jxxl9\" (UID: \"62163913-600f-4a32-adc7-8604e41e1d79\") " pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.562705 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562620 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/62163913-600f-4a32-adc7-8604e41e1d79-signing-key\") pod \"service-ca-658d7b7c78-jxxl9\" (UID: \"62163913-600f-4a32-adc7-8604e41e1d79\") " pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.562705 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562647 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e1b35c-102a-43c8-b556-b4ba82ac1796-config-volume\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.562705 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562679 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.563130 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562721 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.563130 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562751 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwkmc\" (UniqueName: \"kubernetes.io/projected/fb1b5900-c451-4a39-b80e-64ee6581fb10-kube-api-access-xwkmc\") pod \"ingress-canary-8pq82\" (UID: \"fb1b5900-c451-4a39-b80e-64ee6581fb10\") " pod="openshift-ingress-canary/ingress-canary-8pq82" Jul 24 12:53:39.563130 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562775 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglfr\" (UniqueName: \"kubernetes.io/projected/62163913-600f-4a32-adc7-8604e41e1d79-kube-api-access-hglfr\") pod \"service-ca-658d7b7c78-jxxl9\" (UID: \"62163913-600f-4a32-adc7-8604e41e1d79\") " pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.563130 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562800 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnk2\" (UniqueName: \"kubernetes.io/projected/e8e1b35c-102a-43c8-b556-b4ba82ac1796-kube-api-access-swnk2\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.563130 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.562842 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-data-volume\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.563130 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:39.562938 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 12:53:39.563130 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:39.562985 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls podName:e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:40.062970192 +0000 UTC m=+36.066272291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgx2s" (UID: "e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8") : secret "insights-runtime-extractor-tls" not found Jul 24 12:53:39.563130 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.563056 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-crio-socket\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.567454 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.567424 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb1b5900-c451-4a39-b80e-64ee6581fb10-cert\") pod \"ingress-canary-8pq82\" (UID: \"fb1b5900-c451-4a39-b80e-64ee6581fb10\") " pod="openshift-ingress-canary/ingress-canary-8pq82" Jul 24 12:53:39.575359 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.575334 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwkmc\" (UniqueName: \"kubernetes.io/projected/fb1b5900-c451-4a39-b80e-64ee6581fb10-kube-api-access-xwkmc\") pod \"ingress-canary-8pq82\" (UID: \"fb1b5900-c451-4a39-b80e-64ee6581fb10\") " pod="openshift-ingress-canary/ingress-canary-8pq82" Jul 24 12:53:39.575549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.575522 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knw4\" (UniqueName: \"kubernetes.io/projected/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-api-access-2knw4\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:39.578152 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.578127 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:39.663527 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.663485 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e1b35c-102a-43c8-b556-b4ba82ac1796-tmp-dir\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.663724 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.663541 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/62163913-600f-4a32-adc7-8604e41e1d79-signing-cabundle\") pod \"service-ca-658d7b7c78-jxxl9\" (UID: \"62163913-600f-4a32-adc7-8604e41e1d79\") " pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.663724 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.663706 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/62163913-600f-4a32-adc7-8604e41e1d79-signing-key\") pod \"service-ca-658d7b7c78-jxxl9\" (UID: \"62163913-600f-4a32-adc7-8604e41e1d79\") " pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.663840 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.663739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e1b35c-102a-43c8-b556-b4ba82ac1796-config-volume\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.663840 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.663784 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.663840 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.663822 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglfr\" (UniqueName: \"kubernetes.io/projected/62163913-600f-4a32-adc7-8604e41e1d79-kube-api-access-hglfr\") pod \"service-ca-658d7b7c78-jxxl9\" (UID: \"62163913-600f-4a32-adc7-8604e41e1d79\") " pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.663980 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.663855 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnk2\" (UniqueName: \"kubernetes.io/projected/e8e1b35c-102a-43c8-b556-b4ba82ac1796-kube-api-access-swnk2\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.663980 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.663905 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e1b35c-102a-43c8-b556-b4ba82ac1796-tmp-dir\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.664079 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:39.663983 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 12:53:39.664079 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:39.664037 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls podName:e8e1b35c-102a-43c8-b556-b4ba82ac1796 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:40.164020718 +0000 UTC m=+36.167322800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls") pod "dns-default-mw26d" (UID: "e8e1b35c-102a-43c8-b556-b4ba82ac1796") : secret "dns-default-metrics-tls" not found Jul 24 12:53:39.664379 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.664357 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/62163913-600f-4a32-adc7-8604e41e1d79-signing-cabundle\") pod \"service-ca-658d7b7c78-jxxl9\" (UID: \"62163913-600f-4a32-adc7-8604e41e1d79\") " pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.664469 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.664402 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e1b35c-102a-43c8-b556-b4ba82ac1796-config-volume\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.666581 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.666547 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/62163913-600f-4a32-adc7-8604e41e1d79-signing-key\") pod \"service-ca-658d7b7c78-jxxl9\" (UID: \"62163913-600f-4a32-adc7-8604e41e1d79\") " pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.673540 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.673514 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglfr\" (UniqueName: \"kubernetes.io/projected/62163913-600f-4a32-adc7-8604e41e1d79-kube-api-access-hglfr\") pod \"service-ca-658d7b7c78-jxxl9\" (UID: \"62163913-600f-4a32-adc7-8604e41e1d79\") " pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.673672 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.673653 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnk2\" (UniqueName: \"kubernetes.io/projected/e8e1b35c-102a-43c8-b556-b4ba82ac1796-kube-api-access-swnk2\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:39.699815 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.699784 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8pq82" Jul 24 12:53:39.768209 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.768181 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" Jul 24 12:53:39.981583 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.981393 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-jxxl9"] Jul 24 12:53:39.984454 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:39.984425 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8pq82"] Jul 24 12:53:39.991668 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:39.991644 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1b5900_c451_4a39_b80e_64ee6581fb10.slice/crio-3c9b194fc0c9927be3739aa9a9b399a1dc89518c719b551b6c7a46694c2d3935 WatchSource:0}: Error finding container 3c9b194fc0c9927be3739aa9a9b399a1dc89518c719b551b6c7a46694c2d3935: Status 404 returned error can't find the container with id 3c9b194fc0c9927be3739aa9a9b399a1dc89518c719b551b6c7a46694c2d3935 Jul 24 12:53:39.992015 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:39.991992 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62163913_600f_4a32_adc7_8604e41e1d79.slice/crio-26d614c45c495fb22b4fdf193bf1050513a6e9e9f5ef754af5e902e113d5d2c6 WatchSource:0}: Error finding container 26d614c45c495fb22b4fdf193bf1050513a6e9e9f5ef754af5e902e113d5d2c6: Status 404 returned error can't find the container with id 26d614c45c495fb22b4fdf193bf1050513a6e9e9f5ef754af5e902e113d5d2c6 Jul 24 12:53:40.067701 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.067667 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:40.067851 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.067739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:40.067905 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:40.067843 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 12:53:40.067936 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:40.067918 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls podName:e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:41.0679011 +0000 UTC m=+37.071203182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgx2s" (UID: "e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8") : secret "insights-runtime-extractor-tls" not found Jul 24 12:53:40.067997 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:40.067848 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 12:53:40.067997 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:40.067984 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-rbac-proxy-cm podName:e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:41.067972597 +0000 UTC m=+37.071274678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hgx2s" (UID: "e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8") : configmap "kube-rbac-proxy" not found Jul 24 12:53:40.168246 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.168214 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:40.168430 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:40.168404 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 12:53:40.168496 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:40.168484 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls podName:e8e1b35c-102a-43c8-b556-b4ba82ac1796 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:41.168465526 +0000 UTC m=+37.171767609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls") pod "dns-default-mw26d" (UID: "e8e1b35c-102a-43c8-b556-b4ba82ac1796") : secret "dns-default-metrics-tls" not found Jul 24 12:53:40.180513 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.180488 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:40.287038 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.287003 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:40.435320 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.435247 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:40.596352 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.596324 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:53:40.596515 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.596444 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:53:40.596811 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.596627 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:53:40.600205 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.600182 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 12:53:40.600309 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.600206 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 12:53:40.600309 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.600253 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 12:53:40.600309 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.600274 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-txwsv\"" Jul 24 12:53:40.600454 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.600192 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lv5pf\"" Jul 24 12:53:40.600454 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.600183 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 12:53:40.668099 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.668064 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:40.758112 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.757812 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" event={"ID":"62163913-600f-4a32-adc7-8604e41e1d79","Type":"ContainerStarted","Data":"26d614c45c495fb22b4fdf193bf1050513a6e9e9f5ef754af5e902e113d5d2c6"} Jul 24 12:53:40.761598 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.761542 2554 generic.go:356] "Generic (PLEG): container finished" podID="b6af1e2e-1a39-4433-80a8-563a45ebc689" containerID="cb84128b7f770df09fcb4f8c460968fd89eed53f98d17e62e3ee8239adbd6b7e" exitCode=0 Jul 24 12:53:40.761742 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.761597 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" event={"ID":"b6af1e2e-1a39-4433-80a8-563a45ebc689","Type":"ContainerDied","Data":"cb84128b7f770df09fcb4f8c460968fd89eed53f98d17e62e3ee8239adbd6b7e"} Jul 24 12:53:40.763317 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:40.763293 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8pq82" event={"ID":"fb1b5900-c451-4a39-b80e-64ee6581fb10","Type":"ContainerStarted","Data":"3c9b194fc0c9927be3739aa9a9b399a1dc89518c719b551b6c7a46694c2d3935"} Jul 24 12:53:41.077529 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:41.077487 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:41.077726 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:41.077589 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:41.077726 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:41.077670 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 12:53:41.077835 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:41.077767 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls podName:e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:43.077744774 +0000 UTC m=+39.081046871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgx2s" (UID: "e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8") : secret "insights-runtime-extractor-tls" not found Jul 24 12:53:41.091123 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:41.091094 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:41.178305 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:41.178256 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:41.178469 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:41.178397 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 12:53:41.178469 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:41.178463 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls podName:e8e1b35c-102a-43c8-b556-b4ba82ac1796 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:43.17844472 +0000 UTC m=+39.181746817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls") pod "dns-default-mw26d" (UID: "e8e1b35c-102a-43c8-b556-b4ba82ac1796") : secret "dns-default-metrics-tls" not found Jul 24 12:53:41.769321 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:41.769287 2554 generic.go:356] "Generic (PLEG): container finished" podID="b6af1e2e-1a39-4433-80a8-563a45ebc689" containerID="a98dfa59d6b75f7512341771698c364b439a6a71ced629241bbf575f67f51e27" exitCode=0 Jul 24 12:53:41.769827 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:41.769351 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" event={"ID":"b6af1e2e-1a39-4433-80a8-563a45ebc689","Type":"ContainerDied","Data":"a98dfa59d6b75f7512341771698c364b439a6a71ced629241bbf575f67f51e27"} Jul 24 12:53:43.096241 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:43.096211 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:43.096631 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:43.096367 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 12:53:43.096631 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:43.096424 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls podName:e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:47.096407075 +0000 UTC m=+43.099709164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgx2s" (UID: "e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8") : secret "insights-runtime-extractor-tls" not found Jul 24 12:53:43.196997 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:43.196807 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:43.197164 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:43.196966 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 12:53:43.197164 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:43.197161 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls podName:e8e1b35c-102a-43c8-b556-b4ba82ac1796 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:47.197137709 +0000 UTC m=+43.200439797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls") pod "dns-default-mw26d" (UID: "e8e1b35c-102a-43c8-b556-b4ba82ac1796") : secret "dns-default-metrics-tls" not found Jul 24 12:53:43.775594 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:43.775544 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8pq82" event={"ID":"fb1b5900-c451-4a39-b80e-64ee6581fb10","Type":"ContainerStarted","Data":"486262d99b4bdfefbae9786156bd71211b2dfa2a9a037fc8a1fcb8e096bf0def"} Jul 24 12:53:43.777077 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:43.777054 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" event={"ID":"62163913-600f-4a32-adc7-8604e41e1d79","Type":"ContainerStarted","Data":"6e307ebe9acefb12e00a58f2b3bd39c318cfe38e62b66b9bec7e02a4cacad6d6"} Jul 24 12:53:43.780652 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:43.780628 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" event={"ID":"b6af1e2e-1a39-4433-80a8-563a45ebc689","Type":"ContainerStarted","Data":"f3d18e35f45325715d8b00f130e264a1fe4d53caf007f59ecd5b160452064b1f"} Jul 24 12:53:43.795094 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:43.795057 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8pq82" podStartSLOduration=1.7937769810000002 podStartE2EDuration="4.795044905s" totalImagesPullingTime="3.001267924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:43.794001342 +0000 UTC m=+39.797303469" watchObservedRunningTime="2026-07-24 12:53:43.795044905 +0000 UTC m=+39.798347008" Jul 24 12:53:43.812344 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:43.812291 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-jxxl9" podStartSLOduration=1.807297217 podStartE2EDuration="4.812275176s" totalImagesPullingTime="3.004977959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:43.810917925 +0000 UTC m=+39.814220082" watchObservedRunningTime="2026-07-24 12:53:43.812275176 +0000 UTC m=+39.815577281" Jul 24 12:53:43.841519 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:43.841464 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6ljtj" podStartSLOduration=10.498937269 podStartE2EDuration="39.84144455s" totalImagesPullingTime="29.342507281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:43.839665427 +0000 UTC m=+39.842967553" watchObservedRunningTime="2026-07-24 12:53:43.84144455 +0000 UTC m=+39.844746652" Jul 24 12:53:43.908742 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:43.908709 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:44.289687 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:44.289654 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:44.387927 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:44.387896 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:44.518299 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:44.518259 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:44.618452 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:44.618315 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:44.719493 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:44.719462 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:45.004267 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:45.004180 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:46.199707 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:46.199669 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:46.423674 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:46.423647 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:46.588673 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:46.588645 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:46.785686 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:46.785648 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:46.993367 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:46.993282 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:47.128760 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:47.128723 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:47.128902 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:47.128871 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 12:53:47.128970 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:47.128948 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls podName:e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:55.128932586 +0000 UTC m=+51.132234672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hgx2s" (UID: "e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8") : secret "insights-runtime-extractor-tls" not found Jul 24 12:53:47.229611 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:47.229584 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:47.230051 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:47.229749 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 12:53:47.230051 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:53:47.229843 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls podName:e8e1b35c-102a-43c8-b556-b4ba82ac1796 nodeName:}" failed. No retries permitted until 2026-07-24 12:53:55.229820376 +0000 UTC m=+51.233122461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls") pod "dns-default-mw26d" (UID: "e8e1b35c-102a-43c8-b556-b4ba82ac1796") : secret "dns-default-metrics-tls" not found Jul 24 12:53:47.904631 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:47.904596 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:48.004948 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:48.004914 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:48.187747 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:48.187676 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:48.377773 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:48.377742 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:48.618469 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:48.618436 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:48.788140 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:48.788108 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:48.862023 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:48.861989 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:48.982541 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:48.982456 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:49.986480 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:49.986445 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:50.086666 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:50.086633 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:50.187180 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:50.187149 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:50.382269 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:50.382232 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:50.588342 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:50.588312 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:50.784012 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:50.783934 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:50.859136 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:50.859096 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:50.982489 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:50.982458 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:51.643225 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:51.643192 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:52.176064 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:52.176029 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:52.383284 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:52.383250 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:52.580917 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:52.580886 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:52.793957 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:52.793927 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:52.983767 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:52.983676 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:54.177441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:54.177412 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:54.379936 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:54.379903 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:54.576804 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:54.576770 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:54.814600 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:54.814551 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:55.026304 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.026223 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:55.178071 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.178034 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:55.192169 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.192144 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:55.196297 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.196280 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgx2s\" (UID: \"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8\") " pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:55.233959 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.233931 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:55.291832 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.291806 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgx2s" Jul 24 12:53:55.293013 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.292991 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:55.295350 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.295325 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8e1b35c-102a-43c8-b556-b4ba82ac1796-metrics-tls\") pod \"dns-default-mw26d\" (UID: \"e8e1b35c-102a-43c8-b556-b4ba82ac1796\") " pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:55.311630 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.311614 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:55.461139 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.461107 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgx2s"] Jul 24 12:53:55.465597 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:55.465547 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5638423_8fd6_4c2a_bc1b_a3ad4c8943e8.slice/crio-ed8f3dfdf1e28f4a7f3764645238babd360e233788ad23338988dbe416f3c543 WatchSource:0}: Error finding container ed8f3dfdf1e28f4a7f3764645238babd360e233788ad23338988dbe416f3c543: Status 404 returned error can't find the container with id ed8f3dfdf1e28f4a7f3764645238babd360e233788ad23338988dbe416f3c543 Jul 24 12:53:55.488025 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.487987 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mw26d"] Jul 24 12:53:55.490660 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:53:55.490632 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e1b35c_102a_43c8_b556_b4ba82ac1796.slice/crio-4d28e499177a24b0bc20c4db215d5eb4e2834feb6bb0b1c411748bcdc50269a4 WatchSource:0}: Error finding container 4d28e499177a24b0bc20c4db215d5eb4e2834feb6bb0b1c411748bcdc50269a4: Status 404 returned error can't find the container with id 4d28e499177a24b0bc20c4db215d5eb4e2834feb6bb0b1c411748bcdc50269a4 Jul 24 12:53:55.807679 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.807641 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgx2s" event={"ID":"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8","Type":"ContainerStarted","Data":"ed8f3dfdf1e28f4a7f3764645238babd360e233788ad23338988dbe416f3c543"} Jul 24 12:53:55.808931 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:55.808907 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mw26d" event={"ID":"e8e1b35c-102a-43c8-b556-b4ba82ac1796","Type":"ContainerStarted","Data":"4d28e499177a24b0bc20c4db215d5eb4e2834feb6bb0b1c411748bcdc50269a4"} Jul 24 12:53:58.545119 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:58.545081 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:58.819086 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:58.818999 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:58.820133 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:58.820104 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mw26d" event={"ID":"e8e1b35c-102a-43c8-b556-b4ba82ac1796","Type":"ContainerStarted","Data":"419540040df278d9a339b2937bd7b073917d3b85ab3dcc8cf11560d7c4a2fb2c"} Jul 24 12:53:58.820253 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:58.820164 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mw26d" event={"ID":"e8e1b35c-102a-43c8-b556-b4ba82ac1796","Type":"ContainerStarted","Data":"2d910b71ff450bf371085f0ebbfae306df6e2a60e0bb7ac11f85385bd5969819"} Jul 24 12:53:58.820309 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:58.820259 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mw26d" Jul 24 12:53:58.841784 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:58.841734 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mw26d" podStartSLOduration=17.544131133 podStartE2EDuration="19.841716765s" totalImagesPullingTime="2.297585632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:53:58.840339766 +0000 UTC m=+54.843641894" watchObservedRunningTime="2026-07-24 12:53:58.841716765 +0000 UTC m=+54.845018868" Jul 24 12:53:58.881083 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:58.881054 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:59.171927 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:59.171846 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:59.282522 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:59.282488 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:59.419202 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:59.419170 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:59.801265 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:59.801234 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:53:59.902519 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:53:59.902483 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:00.018813 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:00.018775 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:00.221206 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:00.221128 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:00.412028 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:00.411994 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:00.762499 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:00.762465 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:01.833725 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:01.833690 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgx2s" event={"ID":"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8","Type":"ContainerStarted","Data":"96ca8a606460beeea14fb67b2906f0d178907437f5d4c0f60846ad964e60c554"} Jul 24 12:54:03.010890 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.010162 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf"] Jul 24 12:54:03.096253 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.096217 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl"] Jul 24 12:54:03.096443 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.096362 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.099192 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.099166 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 12:54:03.099343 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.099270 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 12:54:03.099343 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.099270 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 12:54:03.099623 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.099604 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 12:54:03.135792 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.135761 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf"] Jul 24 12:54:03.135978 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.135806 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl"] Jul 24 12:54:03.135978 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.135914 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.140256 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.140231 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 12:54:03.140401 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.140274 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 12:54:03.140740 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.140544 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 12:54:03.141086 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.141062 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 12:54:03.151535 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.151506 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35994e8-068c-4ea1-83f9-9680a301a460-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.151707 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.151552 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkx9c\" (UniqueName: \"kubernetes.io/projected/f35994e8-068c-4ea1-83f9-9680a301a460-kube-api-access-mkx9c\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.151707 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.151601 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.151707 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.151648 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgdk\" (UniqueName: \"kubernetes.io/projected/7533c430-d3bf-4850-9eb0-b980dc12899e-kube-api-access-wbgdk\") pod \"klusterlet-addon-workmgr-7b65b5599b-lmltf\" (UID: \"7533c430-d3bf-4850-9eb0-b980dc12899e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.151707 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.151675 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7533c430-d3bf-4850-9eb0-b980dc12899e-tmp\") pod \"klusterlet-addon-workmgr-7b65b5599b-lmltf\" (UID: \"7533c430-d3bf-4850-9eb0-b980dc12899e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.151707 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.151697 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7533c430-d3bf-4850-9eb0-b980dc12899e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b65b5599b-lmltf\" (UID: \"7533c430-d3bf-4850-9eb0-b980dc12899e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.152059 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.151733 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-ca\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.152059 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.151773 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.152059 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.151805 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-hub\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.170903 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.170866 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:03.252381 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.252344 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgdk\" (UniqueName: \"kubernetes.io/projected/7533c430-d3bf-4850-9eb0-b980dc12899e-kube-api-access-wbgdk\") pod \"klusterlet-addon-workmgr-7b65b5599b-lmltf\" (UID: \"7533c430-d3bf-4850-9eb0-b980dc12899e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.252381 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.252386 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7533c430-d3bf-4850-9eb0-b980dc12899e-tmp\") pod \"klusterlet-addon-workmgr-7b65b5599b-lmltf\" (UID: \"7533c430-d3bf-4850-9eb0-b980dc12899e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.252698 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.252415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7533c430-d3bf-4850-9eb0-b980dc12899e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b65b5599b-lmltf\" (UID: \"7533c430-d3bf-4850-9eb0-b980dc12899e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.252698 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.252447 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-ca\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.252698 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.252491 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.252698 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.252524 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-hub\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.252698 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.252585 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35994e8-068c-4ea1-83f9-9680a301a460-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.252698 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.252619 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkx9c\" (UniqueName: \"kubernetes.io/projected/f35994e8-068c-4ea1-83f9-9680a301a460-kube-api-access-mkx9c\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.252698 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.252646 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.253038 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.253003 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7533c430-d3bf-4850-9eb0-b980dc12899e-tmp\") pod \"klusterlet-addon-workmgr-7b65b5599b-lmltf\" (UID: \"7533c430-d3bf-4850-9eb0-b980dc12899e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.253777 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.253751 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35994e8-068c-4ea1-83f9-9680a301a460-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.255810 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.255767 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7533c430-d3bf-4850-9eb0-b980dc12899e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b65b5599b-lmltf\" (UID: \"7533c430-d3bf-4850-9eb0-b980dc12899e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.255901 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.255849 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-ca\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.255950 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.255924 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.256546 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.256524 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-hub\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.256682 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.256663 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f35994e8-068c-4ea1-83f9-9680a301a460-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.264103 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.264021 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkx9c\" (UniqueName: \"kubernetes.io/projected/f35994e8-068c-4ea1-83f9-9680a301a460-kube-api-access-mkx9c\") pod \"cluster-proxy-proxy-agent-5bc745bc44-7sdnl\" (UID: \"f35994e8-068c-4ea1-83f9-9680a301a460\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.264390 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.264354 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgdk\" (UniqueName: \"kubernetes.io/projected/7533c430-d3bf-4850-9eb0-b980dc12899e-kube-api-access-wbgdk\") pod \"klusterlet-addon-workmgr-7b65b5599b-lmltf\" (UID: \"7533c430-d3bf-4850-9eb0-b980dc12899e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.266890 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.266819 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:03.372726 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.372685 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:03.409585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.409529 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:03.458441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.458392 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" Jul 24 12:54:03.465212 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.465186 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:03.565092 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:03.565056 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:04.639325 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:04.639292 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf"] Jul 24 12:54:04.644656 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:54:04.644623 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7533c430_d3bf_4850_9eb0_b980dc12899e.slice/crio-3bd724422ed603abfb26a5fa72d03c5287ea740ad5e8d504a157cc6e0f9bc0c3 WatchSource:0}: Error finding container 3bd724422ed603abfb26a5fa72d03c5287ea740ad5e8d504a157cc6e0f9bc0c3: Status 404 returned error can't find the container with id 3bd724422ed603abfb26a5fa72d03c5287ea740ad5e8d504a157cc6e0f9bc0c3 Jul 24 12:54:04.661094 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:04.661072 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl"] Jul 24 12:54:04.663861 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:54:04.663824 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf35994e8_068c_4ea1_83f9_9680a301a460.slice/crio-6a3e63fe1a656da30014077dff58a67200bd5cbb5946db37f045fe2a1a9e427f WatchSource:0}: Error finding container 6a3e63fe1a656da30014077dff58a67200bd5cbb5946db37f045fe2a1a9e427f: Status 404 returned error can't find the container with id 6a3e63fe1a656da30014077dff58a67200bd5cbb5946db37f045fe2a1a9e427f Jul 24 12:54:04.844768 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:04.844729 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" event={"ID":"f35994e8-068c-4ea1-83f9-9680a301a460","Type":"ContainerStarted","Data":"6a3e63fe1a656da30014077dff58a67200bd5cbb5946db37f045fe2a1a9e427f"} Jul 24 12:54:04.846362 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:04.846335 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgx2s" event={"ID":"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8","Type":"ContainerStarted","Data":"1ab737117e0a9687b30fbf07989f9fc0b96e9150a7fe2224b939f2822c56e5ab"} Jul 24 12:54:04.847361 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:04.847341 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" event={"ID":"7533c430-d3bf-4850-9eb0-b980dc12899e","Type":"ContainerStarted","Data":"3bd724422ed603abfb26a5fa72d03c5287ea740ad5e8d504a157cc6e0f9bc0c3"} Jul 24 12:54:05.994271 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:05.994215 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:06.097420 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:06.097388 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:06.200609 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:06.200550 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:06.283648 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:06.283547 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:06.348033 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:06.347943 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:06.415158 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:06.415091 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:06.761957 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:06.761927 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtlpz" Jul 24 12:54:06.896917 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:06.896892 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:07.023248 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:07.023145 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:08.136369 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:08.136277 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:08.236506 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:08.236471 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:08.825599 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:08.825547 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mw26d" Jul 24 12:54:10.226440 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.226402 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:54:10.229588 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.229508 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 12:54:10.239878 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.239823 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415e7d0e-e69a-46d7-b32c-ad60c5487fab-metrics-certs\") pod \"network-metrics-daemon-jcst2\" (UID: \"415e7d0e-e69a-46d7-b32c-ad60c5487fab\") " pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:54:10.324143 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.324110 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-txwsv\"" Jul 24 12:54:10.327185 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.327155 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:54:10.327337 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.327209 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:54:10.329928 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.329899 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 12:54:10.330070 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.329911 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 12:54:10.330943 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.330891 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcst2" Jul 24 12:54:10.340013 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.339987 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 12:54:10.340134 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.340105 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbea5b3b-f397-4df0-a942-9b238c3d15aa-original-pull-secret\") pod \"global-pull-secret-syncer-ltm9n\" (UID: \"cbea5b3b-f397-4df0-a942-9b238c3d15aa\") " pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:54:10.350904 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.350881 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7sb\" (UniqueName: \"kubernetes.io/projected/3e72bde8-aa26-4b4a-9929-be0a7b6594d4-kube-api-access-mj7sb\") pod \"network-check-target-jwxch\" (UID: \"3e72bde8-aa26-4b4a-9929-be0a7b6594d4\") " pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:54:10.612021 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.611991 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lv5pf\"" Jul 24 12:54:10.616366 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.616346 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ltm9n" Jul 24 12:54:10.619473 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:10.619456 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:54:11.098523 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:11.098484 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:11.202804 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:11.202766 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:11.286232 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:11.286182 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:11.398990 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:11.398896 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:12.059116 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.059065 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:12.122129 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.122095 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:12.191058 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.190976 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jcst2"] Jul 24 12:54:12.195544 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:54:12.195457 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415e7d0e_e69a_46d7_b32c_ad60c5487fab.slice/crio-6ad018c0dad90a978095daa3c9934db9ae3c8edf3fae6a6086858389328c802d WatchSource:0}: Error finding container 6ad018c0dad90a978095daa3c9934db9ae3c8edf3fae6a6086858389328c802d: Status 404 returned error can't find the container with id 6ad018c0dad90a978095daa3c9934db9ae3c8edf3fae6a6086858389328c802d Jul 24 12:54:12.418067 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.418031 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jwxch"] Jul 24 12:54:12.420475 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.420451 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ltm9n"] Jul 24 12:54:12.421205 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:54:12.421182 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e72bde8_aa26_4b4a_9929_be0a7b6594d4.slice/crio-8cfead31534f9eca083b674ef858df748c8f893f91700325d6f673bf4df71c12 WatchSource:0}: Error finding container 8cfead31534f9eca083b674ef858df748c8f893f91700325d6f673bf4df71c12: Status 404 returned error can't find the container with id 8cfead31534f9eca083b674ef858df748c8f893f91700325d6f673bf4df71c12 Jul 24 12:54:12.423412 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:54:12.423388 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbea5b3b_f397_4df0_a942_9b238c3d15aa.slice/crio-67b56269077fe8fc7a00f4cd7781517335e724885599d84554e93ff87beb2caf WatchSource:0}: Error finding container 67b56269077fe8fc7a00f4cd7781517335e724885599d84554e93ff87beb2caf: Status 404 returned error can't find the container with id 67b56269077fe8fc7a00f4cd7781517335e724885599d84554e93ff87beb2caf Jul 24 12:54:12.884444 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.884407 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" event={"ID":"f35994e8-068c-4ea1-83f9-9680a301a460","Type":"ContainerStarted","Data":"3ba5c7b2d71d6a4e48d71e52b69161a78258b1d44350ec198eb4d8e0a6cc2233"} Jul 24 12:54:12.884444 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.884450 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" event={"ID":"f35994e8-068c-4ea1-83f9-9680a301a460","Type":"ContainerStarted","Data":"bb2f354f0bd672b92bd5507c5a06adc8a73c8ce132fae3f33f935b66b5009c29"} Jul 24 12:54:12.884713 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.884462 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" event={"ID":"f35994e8-068c-4ea1-83f9-9680a301a460","Type":"ContainerStarted","Data":"d8cce387a75e557f365a6726accc9cd7fc2c90105e9bdb8eab8bde8b08858696"} Jul 24 12:54:12.885901 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.885845 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ltm9n" event={"ID":"cbea5b3b-f397-4df0-a942-9b238c3d15aa","Type":"ContainerStarted","Data":"67b56269077fe8fc7a00f4cd7781517335e724885599d84554e93ff87beb2caf"} Jul 24 12:54:12.888249 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.888106 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgx2s" event={"ID":"e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8","Type":"ContainerStarted","Data":"c49e50a9f2507438ea64b0dc1325fa6e1c68dd38b011a5e270fc76fa98d03893"} Jul 24 12:54:12.889962 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.889895 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jcst2" event={"ID":"415e7d0e-e69a-46d7-b32c-ad60c5487fab","Type":"ContainerStarted","Data":"6ad018c0dad90a978095daa3c9934db9ae3c8edf3fae6a6086858389328c802d"} Jul 24 12:54:12.892230 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.892206 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" event={"ID":"7533c430-d3bf-4850-9eb0-b980dc12899e","Type":"ContainerStarted","Data":"64bb50b1ef7688e5cd48cf22b751841556a6ecf7fee34e5dd07443f99a7cb05c"} Jul 24 12:54:12.892768 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.892716 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:12.893791 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.893769 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jwxch" event={"ID":"3e72bde8-aa26-4b4a-9929-be0a7b6594d4","Type":"ContainerStarted","Data":"8cfead31534f9eca083b674ef858df748c8f893f91700325d6f673bf4df71c12"} Jul 24 12:54:12.894576 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.894544 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" Jul 24 12:54:12.904636 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.904588 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bc745bc44-7sdnl" podStartSLOduration=3.529744303 podStartE2EDuration="10.904555893s" totalImagesPullingTime="7.37481159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:12.90329227 +0000 UTC m=+68.906594404" watchObservedRunningTime="2026-07-24 12:54:12.904555893 +0000 UTC m=+68.907857996" Jul 24 12:54:12.941792 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:12.941440 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hgx2s" podStartSLOduration=17.504683634 podStartE2EDuration="33.941419814s" totalImagesPullingTime="16.43673618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:12.922746966 +0000 UTC m=+68.926049067" watchObservedRunningTime="2026-07-24 12:54:12.941419814 +0000 UTC m=+68.944721919" Jul 24 12:54:13.253281 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:13.253200 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:13.356089 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:13.355731 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:13.517190 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:13.517105 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:13.901606 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:13.901550 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jcst2" event={"ID":"415e7d0e-e69a-46d7-b32c-ad60c5487fab","Type":"ContainerStarted","Data":"8f91f49dad248b73e4447b4d568522e163c548386af10580e8d7513d35bd49ed"} Jul 24 12:54:13.901801 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:13.901666 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jcst2" event={"ID":"415e7d0e-e69a-46d7-b32c-ad60c5487fab","Type":"ContainerStarted","Data":"bafa09ad6975331dfc78f7877e9d7409b7689ab66805f5ddce0093c5c6f7b180"} Jul 24 12:54:13.921922 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:13.921863 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b65b5599b-lmltf" podStartSLOduration=4.529402153 podStartE2EDuration="11.921842155s" totalImagesPullingTime="7.392440002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:12.941722071 +0000 UTC m=+68.945024199" watchObservedRunningTime="2026-07-24 12:54:13.921842155 +0000 UTC m=+69.925144260" Jul 24 12:54:13.922524 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:13.922444 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jcst2" podStartSLOduration=68.559610962 podStartE2EDuration="1m9.922432143s" totalImagesPullingTime="1.362821181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:13.921438971 +0000 UTC m=+69.924741077" watchObservedRunningTime="2026-07-24 12:54:13.922432143 +0000 UTC m=+69.925734250" Jul 24 12:54:14.026752 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:14.026705 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:14.090746 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:14.090701 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:15.405113 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.405076 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:15.564188 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.564154 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:15.671882 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.671812 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:15.715119 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.714072 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rz6fc"] Jul 24 12:54:15.717339 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.717317 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.719955 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.719857 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9srnh\"" Jul 24 12:54:15.720096 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.720054 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 12:54:15.721202 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.720962 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 12:54:15.721202 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.721001 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 12:54:15.721202 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.721017 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 12:54:15.721202 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.721106 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 12:54:15.721417 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.721401 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 12:54:15.770380 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.770345 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-textfile\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.770585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.770390 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-sys\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.770585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.770425 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-root\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.770585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.770453 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.770585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.770492 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-tls\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.770585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.770519 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rqw6\" (UniqueName: \"kubernetes.io/projected/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-kube-api-access-2rqw6\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.770585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.770544 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.770902 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.770596 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-wtmp\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.770902 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.770625 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-metrics-client-ca\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.773449 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.773395 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:15.862164 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.862094 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:15.871041 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871009 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-textfile\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871041 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871050 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-sys\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871229 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871123 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-sys\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871463 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871444 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-textfile\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871501 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871490 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-root\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871535 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871523 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871604 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871593 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-tls\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871673 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871623 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rqw6\" (UniqueName: \"kubernetes.io/projected/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-kube-api-access-2rqw6\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871673 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871651 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871789 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871688 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-wtmp\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.871789 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.871720 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-metrics-client-ca\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.873675 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.872204 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-metrics-client-ca\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.873675 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.872262 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-root\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.873675 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.872662 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.873675 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:54:15.872750 2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 12:54:15.873675 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:54:15.872805 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-tls podName:e8a9119e-fb00-42e5-8e4b-71039aeaf40a nodeName:}" failed. No retries permitted until 2026-07-24 12:54:16.372786697 +0000 UTC m=+72.376088786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-tls") pod "node-exporter-rz6fc" (UID: "e8a9119e-fb00-42e5-8e4b-71039aeaf40a") : secret "node-exporter-tls" not found Jul 24 12:54:15.873675 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.873620 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-wtmp\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.876773 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.876750 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.886461 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.886438 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rqw6\" (UniqueName: \"kubernetes.io/projected/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-kube-api-access-2rqw6\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:15.961231 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:15.961140 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:16.061938 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.061903 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:16.149742 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.149710 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:16.247971 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.247894 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:16.375685 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.375648 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-tls\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:16.378551 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.378518 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8a9119e-fb00-42e5-8e4b-71039aeaf40a-node-exporter-tls\") pod \"node-exporter-rz6fc\" (UID: \"e8a9119e-fb00-42e5-8e4b-71039aeaf40a\") " pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:16.630698 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.630454 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rz6fc" Jul 24 12:54:16.672485 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.672411 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:16.746990 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.746944 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:16.827919 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.827882 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:16.924702 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:16.924611 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:17.588902 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:54:17.588877 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a9119e_fb00_42e5_8e4b_71039aeaf40a.slice/crio-a89397e46b0165b9cee079e584d51ade427161333b44a940ab3edd9a66b52366 WatchSource:0}: Error finding container a89397e46b0165b9cee079e584d51ade427161333b44a940ab3edd9a66b52366: Status 404 returned error can't find the container with id a89397e46b0165b9cee079e584d51ade427161333b44a940ab3edd9a66b52366 Jul 24 12:54:17.919282 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:17.919181 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jwxch" event={"ID":"3e72bde8-aa26-4b4a-9929-be0a7b6594d4","Type":"ContainerStarted","Data":"d1b85bb3cfd641b719bd940f84fcb63deba5008255fd5cd0083653e4dc7c2aa4"} Jul 24 12:54:17.919282 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:17.919245 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:54:17.920756 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:17.920729 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ltm9n" event={"ID":"cbea5b3b-f397-4df0-a942-9b238c3d15aa","Type":"ContainerStarted","Data":"0dfd5f29277fae5a863f4a57f608e11796112750d2fcdb9ea5aa20da8da591bd"} Jul 24 12:54:17.921976 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:17.921960 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rz6fc" event={"ID":"e8a9119e-fb00-42e5-8e4b-71039aeaf40a","Type":"ContainerStarted","Data":"a89397e46b0165b9cee079e584d51ade427161333b44a940ab3edd9a66b52366"} Jul 24 12:54:17.938236 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:17.938185 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jwxch" podStartSLOduration=68.773196811 podStartE2EDuration="1m13.93817085s" totalImagesPullingTime="5.164974039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:17.937150363 +0000 UTC m=+73.940452469" watchObservedRunningTime="2026-07-24 12:54:17.93817085 +0000 UTC m=+73.941472953" Jul 24 12:54:17.953864 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:17.953812 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ltm9n" podStartSLOduration=68.775680155 podStartE2EDuration="1m13.953798269s" totalImagesPullingTime="5.178118114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:17.952994185 +0000 UTC m=+73.956296290" watchObservedRunningTime="2026-07-24 12:54:17.953798269 +0000 UTC m=+73.957100374" Jul 24 12:54:17.994995 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:17.994963 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:18.072088 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:18.072052 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:18.630042 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:18.630014 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:18.718267 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:18.718238 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:18.809636 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:18.809612 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:18.881713 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:18.881690 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:18.927399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:18.927367 2554 generic.go:356] "Generic (PLEG): container finished" podID="e8a9119e-fb00-42e5-8e4b-71039aeaf40a" containerID="488ccc3829bc525f250562b36c907f90596d32fcf4ae28d3d69b35e6d56cd9eb" exitCode=0 Jul 24 12:54:18.927827 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:18.927446 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rz6fc" event={"ID":"e8a9119e-fb00-42e5-8e4b-71039aeaf40a","Type":"ContainerDied","Data":"488ccc3829bc525f250562b36c907f90596d32fcf4ae28d3d69b35e6d56cd9eb"} Jul 24 12:54:19.933668 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:19.933629 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rz6fc" event={"ID":"e8a9119e-fb00-42e5-8e4b-71039aeaf40a","Type":"ContainerStarted","Data":"82d7b3ad8a4467b8c7cc046543294fe0ce78c7b2e83f5b26eb39953d3b024566"} Jul 24 12:54:19.934102 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:19.933712 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rz6fc" event={"ID":"e8a9119e-fb00-42e5-8e4b-71039aeaf40a","Type":"ContainerStarted","Data":"b3953a71d792d1c75fa9364c4aaea1c2407b10d2a5f506f1096e36325b04e7bf"} Jul 24 12:54:19.957519 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:19.957468 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rz6fc" podStartSLOduration=3.996809349 podStartE2EDuration="4.957453085s" totalImagesPullingTime="883.500663ms" totalInitContainerRuntime="77.143073ms" isStatefulPod=true podCreationTimestamp="2026-07-24 12:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:19.956682155 +0000 UTC m=+75.959984251" watchObservedRunningTime="2026-07-24 12:54:19.957453085 +0000 UTC m=+75.960755188" Jul 24 12:54:20.313055 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.313003 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:20.417830 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.417787 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:20.474765 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.474734 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g"] Jul 24 12:54:20.476607 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.476592 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:20.479664 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.479644 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bkxfx\"" Jul 24 12:54:20.480008 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.479993 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 24 12:54:20.486824 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.486804 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g"] Jul 24 12:54:20.494654 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.494636 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:20.611510 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.611420 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34abfc3c-d51e-4715-84fc-e7dc20268fb7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-9dp5g\" (UID: \"34abfc3c-d51e-4715-84fc-e7dc20268fb7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:20.611510 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.611469 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wt9\" (UniqueName: \"kubernetes.io/projected/34abfc3c-d51e-4715-84fc-e7dc20268fb7-kube-api-access-77wt9\") pod \"monitoring-plugin-68b7fc9574-9dp5g\" (UID: \"34abfc3c-d51e-4715-84fc-e7dc20268fb7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:20.712587 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.712523 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34abfc3c-d51e-4715-84fc-e7dc20268fb7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-9dp5g\" (UID: \"34abfc3c-d51e-4715-84fc-e7dc20268fb7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:20.712765 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.712600 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wt9\" (UniqueName: \"kubernetes.io/projected/34abfc3c-d51e-4715-84fc-e7dc20268fb7-kube-api-access-77wt9\") pod \"monitoring-plugin-68b7fc9574-9dp5g\" (UID: \"34abfc3c-d51e-4715-84fc-e7dc20268fb7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:20.712765 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:54:20.712687 2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 24 12:54:20.712847 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:54:20.712768 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34abfc3c-d51e-4715-84fc-e7dc20268fb7-monitoring-plugin-cert podName:34abfc3c-d51e-4715-84fc-e7dc20268fb7 nodeName:}" failed. No retries permitted until 2026-07-24 12:54:21.212748094 +0000 UTC m=+77.216050196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/34abfc3c-d51e-4715-84fc-e7dc20268fb7-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-9dp5g" (UID: "34abfc3c-d51e-4715-84fc-e7dc20268fb7") : secret "monitoring-plugin-cert" not found Jul 24 12:54:20.722438 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.722415 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wt9\" (UniqueName: \"kubernetes.io/projected/34abfc3c-d51e-4715-84fc-e7dc20268fb7-kube-api-access-77wt9\") pod \"monitoring-plugin-68b7fc9574-9dp5g\" (UID: \"34abfc3c-d51e-4715-84fc-e7dc20268fb7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:20.999535 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:20.999455 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:21.217047 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.217009 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34abfc3c-d51e-4715-84fc-e7dc20268fb7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-9dp5g\" (UID: \"34abfc3c-d51e-4715-84fc-e7dc20268fb7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:21.219591 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.219551 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34abfc3c-d51e-4715-84fc-e7dc20268fb7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-9dp5g\" (UID: \"34abfc3c-d51e-4715-84fc-e7dc20268fb7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:21.386151 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.386109 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:21.512117 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.512083 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g"] Jul 24 12:54:21.515052 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:54:21.515027 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34abfc3c_d51e_4715_84fc_e7dc20268fb7.slice/crio-e5ab63cf3ddd87129ca4ddc9308bc3ad9e155b1184e3b2e7dd9c31b63ad22475 WatchSource:0}: Error finding container e5ab63cf3ddd87129ca4ddc9308bc3ad9e155b1184e3b2e7dd9c31b63ad22475: Status 404 returned error can't find the container with id e5ab63cf3ddd87129ca4ddc9308bc3ad9e155b1184e3b2e7dd9c31b63ad22475 Jul 24 12:54:21.940788 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.940757 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" event={"ID":"34abfc3c-d51e-4715-84fc-e7dc20268fb7","Type":"ContainerStarted","Data":"e5ab63cf3ddd87129ca4ddc9308bc3ad9e155b1184e3b2e7dd9c31b63ad22475"} Jul 24 12:54:21.961745 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.961711 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:21.966850 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.966827 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 12:54:21.969133 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.969113 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:21.971688 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.971665 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 12:54:21.971961 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.971941 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 12:54:21.972373 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.972355 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 12:54:21.972462 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.972418 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 12:54:21.972513 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.972362 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 12:54:21.972757 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.972736 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 12:54:21.973490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.973001 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4c9hfqbcbm23r\"" Jul 24 12:54:21.973490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.973018 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 12:54:21.973490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.973025 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 12:54:21.973490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.973004 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 12:54:21.973490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.973050 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j5kfz\"" Jul 24 12:54:21.973490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.973325 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 12:54:21.973490 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.973476 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 12:54:21.973818 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.973607 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 12:54:21.974085 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.974066 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 12:54:21.976167 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.975956 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 12:54:21.981057 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.981037 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 12:54:21.988015 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:21.987992 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 12:54:22.021903 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.021868 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.021916 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.021944 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022052 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022083 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022112 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022172 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022213 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022240 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022256 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022273 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-config\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022323 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-config-out\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022347 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-web-config\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022387 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022404 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhskp\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-kube-api-access-jhskp\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022422 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022525 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022575 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022605 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.022817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.022625 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.123539 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123505 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.123539 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123539 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.123788 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123579 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.123788 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123597 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.123788 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123612 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-config\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.123934 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123810 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-config-out\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.123934 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123853 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-web-config\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.123934 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123893 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.123934 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123916 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhskp\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-kube-api-access-jhskp\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124126 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123943 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124126 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123968 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124126 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.123996 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124126 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.124025 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124126 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.124049 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124126 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.124089 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124126 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.124114 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.124140 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.124266 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.124359 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.124390 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.124817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.124634 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.125386 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.125363 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.127597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.126447 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.127597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.126897 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.127597 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.127441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.127824 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.127619 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.129810 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.129851 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.130226 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.130427 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-config\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.130756 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-config-out\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.130975 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.131322 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.132226 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.132716 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.132888 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.132996 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.133109 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-web-config\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.135714 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.133649 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.137623 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.137601 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhskp\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-kube-api-access-jhskp\") pod \"prometheus-k8s-0\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.281884 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.281790 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:22.450203 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.450171 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 12:54:22.708813 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:54:22.708782 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60344f41_064c_4bba_b20f_2695d092c46f.slice/crio-6c481ec33d4b93fa83e92162da3a680134dbfd25abedfb0c63e893c542f33671 WatchSource:0}: Error finding container 6c481ec33d4b93fa83e92162da3a680134dbfd25abedfb0c63e893c542f33671: Status 404 returned error can't find the container with id 6c481ec33d4b93fa83e92162da3a680134dbfd25abedfb0c63e893c542f33671 Jul 24 12:54:22.945613 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.945578 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" event={"ID":"34abfc3c-d51e-4715-84fc-e7dc20268fb7","Type":"ContainerStarted","Data":"614261766eb834817f6d6ba9d047bf9cd7cef08df875909b9e1c75404250fda7"} Jul 24 12:54:22.945788 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.945642 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:22.946824 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.946792 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerStarted","Data":"6c481ec33d4b93fa83e92162da3a680134dbfd25abedfb0c63e893c542f33671"} Jul 24 12:54:22.950330 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.950308 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" Jul 24 12:54:22.964250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:22.964169 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9dp5g" podStartSLOduration=1.7468547490000002 podStartE2EDuration="2.964154828s" totalImagesPullingTime="1.217300079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:22.963078694 +0000 UTC m=+78.966380826" watchObservedRunningTime="2026-07-24 12:54:22.964154828 +0000 UTC m=+78.967456933" Jul 24 12:54:23.953705 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:23.953620 2554 generic.go:356] "Generic (PLEG): container finished" podID="60344f41-064c-4bba-b20f-2695d092c46f" containerID="ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c" exitCode=0 Jul 24 12:54:23.954063 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:23.953711 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerDied","Data":"ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c"} Jul 24 12:54:24.249588 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:24.249499 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:24.320019 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:24.319992 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:26.968824 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:26.968689 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerStarted","Data":"2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee"} Jul 24 12:54:26.968824 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:26.968777 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerStarted","Data":"773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011"} Jul 24 12:54:26.992800 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:26.992713 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:27.072404 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:27.072375 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:28.979814 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:28.979779 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerStarted","Data":"b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08"} Jul 24 12:54:28.980228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:28.979843 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerStarted","Data":"7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0"} Jul 24 12:54:28.980228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:28.979868 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerStarted","Data":"d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c"} Jul 24 12:54:28.980228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:28.979893 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerStarted","Data":"5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4"} Jul 24 12:54:29.015661 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:29.015608 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.721521681 podStartE2EDuration="8.015588703s" totalImagesPullingTime="5.277206781s" totalInitContainerRuntime="16.860241ms" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:29.013418199 +0000 UTC m=+85.016720314" watchObservedRunningTime="2026-07-24 12:54:29.015588703 +0000 UTC m=+85.018891000" Jul 24 12:54:29.016271 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:29.016242 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:29.122438 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:29.122402 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:29.221024 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:29.220992 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:29.321870 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:29.321828 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:31.109316 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:31.109285 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:31.211512 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:31.211475 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:32.255958 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:32.255923 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:32.282629 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:32.282598 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:54:32.350042 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:32.350012 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:32.449912 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:32.449874 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:36.007875 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:36.007841 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:36.107509 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:36.107476 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:37.473990 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:37.473958 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:37.579911 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:37.579881 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:37.667811 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:37.667758 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:38.100825 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:38.100793 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:38.200799 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:38.200775 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:40.182070 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:40.182036 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:40.284040 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:40.284003 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:41.697015 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:41.696985 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:48.930631 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:48.930601 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jwxch" Jul 24 12:54:52.552776 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:52.552742 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:52.657014 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:52.656956 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:55.176536 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:55.176500 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:54:55.241256 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:54:55.241222 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:22.282052 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:22.282009 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:22.301998 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:22.301967 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:23.188424 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:23.188391 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:31.806400 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:31.806360 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:31.915493 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:31.915457 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:38.246516 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:38.246482 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:38.338844 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:38.338811 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:40.486825 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:40.486787 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 12:55:40.487245 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:40.487085 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="prometheus" containerID="cri-o://773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011" gracePeriod=600 Jul 24 12:55:40.487245 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:40.487117 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy" containerID="cri-o://7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0" gracePeriod=600 Jul 24 12:55:40.487245 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:40.487121 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="thanos-sidecar" containerID="cri-o://5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4" gracePeriod=600 Jul 24 12:55:40.487245 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:40.487134 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy-web" containerID="cri-o://d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c" gracePeriod=600 Jul 24 12:55:40.487245 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:40.487178 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="config-reloader" containerID="cri-o://2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee" gracePeriod=600 Jul 24 12:55:40.487245 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:40.487188 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08" gracePeriod=600 Jul 24 12:55:41.236283 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236245 2554 generic.go:356] "Generic (PLEG): container finished" podID="60344f41-064c-4bba-b20f-2695d092c46f" containerID="b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08" exitCode=0 Jul 24 12:55:41.236283 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236270 2554 generic.go:356] "Generic (PLEG): container finished" podID="60344f41-064c-4bba-b20f-2695d092c46f" containerID="7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0" exitCode=0 Jul 24 12:55:41.236283 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236279 2554 generic.go:356] "Generic (PLEG): container finished" podID="60344f41-064c-4bba-b20f-2695d092c46f" containerID="5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4" exitCode=0 Jul 24 12:55:41.236283 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236289 2554 generic.go:356] "Generic (PLEG): container finished" podID="60344f41-064c-4bba-b20f-2695d092c46f" containerID="2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee" exitCode=0 Jul 24 12:55:41.236580 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236296 2554 generic.go:356] "Generic (PLEG): container finished" podID="60344f41-064c-4bba-b20f-2695d092c46f" containerID="773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011" exitCode=0 Jul 24 12:55:41.236580 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236312 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerDied","Data":"b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08"} Jul 24 12:55:41.236580 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236377 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerDied","Data":"7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0"} Jul 24 12:55:41.236580 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236404 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerDied","Data":"5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4"} Jul 24 12:55:41.236580 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236430 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerDied","Data":"2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee"} Jul 24 12:55:41.236580 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.236452 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerDied","Data":"773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011"} Jul 24 12:55:41.740137 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.740106 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:41.833962 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.833880 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-trusted-ca-bundle\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.833962 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.833913 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-tls-assets\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.833962 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.833953 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-1\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.833971 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-config-out\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.833994 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-web-config\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834010 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-kubelet-serving-ca-bundle\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834025 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhskp\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-kube-api-access-jhskp\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834050 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-grpc-tls\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834076 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-2\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834102 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-config\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834129 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-serving-certs-ca-bundle\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834155 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-thanos-prometheus-http-client-file\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834183 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834228 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834209 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-metrics-client-ca\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834781 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834239 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-db\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834781 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834267 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-tls\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834781 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834300 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834781 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834330 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 12:55:41.834781 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834339 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 12:55:41.834781 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834349 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-metrics-client-certs\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834781 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834419 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-0\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.834781 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834450 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-kube-rbac-proxy\") pod \"60344f41-064c-4bba-b20f-2695d092c46f\" (UID: \"60344f41-064c-4bba-b20f-2695d092c46f\") " Jul 24 12:55:41.835337 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834815 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.835337 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.834835 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.835337 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.835147 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 12:55:41.835514 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.835489 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 12:55:41.837465 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.837260 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 12:55:41.837795 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.837651 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-config-out" (OuterVolumeSpecName: "config-out") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 12:55:41.837795 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.837732 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 12:55:41.838466 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.838035 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 12:55:41.838466 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.838196 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 12:55:41.838466 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.838402 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 12:55:41.838907 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.838850 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 12:55:41.839006 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.838964 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 12:55:41.840026 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.839323 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-kube-api-access-jhskp" (OuterVolumeSpecName: "kube-api-access-jhskp") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "kube-api-access-jhskp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 12:55:41.840026 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.839944 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 12:55:41.840026 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.839961 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 12:55:41.840506 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.840478 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-config" (OuterVolumeSpecName: "config") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 12:55:41.840630 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.840511 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 12:55:41.841403 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.841377 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 12:55:41.841755 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.841735 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 12:55:41.850585 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.850534 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-web-config" (OuterVolumeSpecName: "web-config") pod "60344f41-064c-4bba-b20f-2695d092c46f" (UID: "60344f41-064c-4bba-b20f-2695d092c46f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 12:55:41.935172 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935144 2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-config-out\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935172 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935169 2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-web-config\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935172 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935179 2554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935189 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhskp\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-kube-api-access-jhskp\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935199 2554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-grpc-tls\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935207 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935215 2554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-config\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935222 2554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935232 2554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935241 2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935251 2554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-configmap-metrics-client-ca\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935262 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-db\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935271 2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935279 2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935289 2554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-metrics-client-certs\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935297 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60344f41-064c-4bba-b20f-2695d092c46f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935305 2554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60344f41-064c-4bba-b20f-2695d092c46f-secret-kube-rbac-proxy\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:41.935399 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:41.935312 2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60344f41-064c-4bba-b20f-2695d092c46f-tls-assets\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 12:55:42.243248 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.243160 2554 generic.go:356] "Generic (PLEG): container finished" podID="60344f41-064c-4bba-b20f-2695d092c46f" containerID="d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c" exitCode=0 Jul 24 12:55:42.243248 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.243216 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerDied","Data":"d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c"} Jul 24 12:55:42.243408 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.243253 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.243408 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.243305 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60344f41-064c-4bba-b20f-2695d092c46f","Type":"ContainerDied","Data":"6c481ec33d4b93fa83e92162da3a680134dbfd25abedfb0c63e893c542f33671"} Jul 24 12:55:42.243408 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.243340 2554 scope.go:117] "RemoveContainer" containerID="b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08" Jul 24 12:55:42.251409 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.251393 2554 scope.go:117] "RemoveContainer" containerID="7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0" Jul 24 12:55:42.258201 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.258180 2554 scope.go:117] "RemoveContainer" containerID="d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c" Jul 24 12:55:42.264854 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.264832 2554 scope.go:117] "RemoveContainer" containerID="5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4" Jul 24 12:55:42.272261 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.272189 2554 scope.go:117] "RemoveContainer" containerID="2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee" Jul 24 12:55:42.272688 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.272672 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 12:55:42.279366 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.279344 2554 scope.go:117] "RemoveContainer" containerID="773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011" Jul 24 12:55:42.280427 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.280406 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 12:55:42.287139 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.287121 2554 scope.go:117] "RemoveContainer" containerID="ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c" Jul 24 12:55:42.293847 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.293830 2554 scope.go:117] "RemoveContainer" containerID="b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08" Jul 24 12:55:42.294166 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:55:42.294146 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08\": container with ID starting with b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08 not found: ID does not exist" containerID="b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08" Jul 24 12:55:42.294213 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.294179 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08"} err="failed to get container status \"b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08\": rpc error: code = NotFound desc = could not find container \"b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08\": container with ID starting with b1b713d7c3b6d4c304b1d459f266435481a9b180b88ca915605acd0a6456ff08 not found: ID does not exist" Jul 24 12:55:42.294262 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.294216 2554 scope.go:117] "RemoveContainer" containerID="7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0" Jul 24 12:55:42.294440 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:55:42.294425 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0\": container with ID starting with 7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0 not found: ID does not exist" containerID="7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0" Jul 24 12:55:42.294483 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.294444 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0"} err="failed to get container status \"7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0\": rpc error: code = NotFound desc = could not find container \"7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0\": container with ID starting with 7d903d0b46a8156d39ba5d660d0bc02812b1c385d44b61157db1525215facae0 not found: ID does not exist" Jul 24 12:55:42.294483 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.294459 2554 scope.go:117] "RemoveContainer" containerID="d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c" Jul 24 12:55:42.294683 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:55:42.294667 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c\": container with ID starting with d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c not found: ID does not exist" containerID="d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c" Jul 24 12:55:42.294771 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.294688 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c"} err="failed to get container status \"d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c\": rpc error: code = NotFound desc = could not find container \"d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c\": container with ID starting with d3da11f28e6491c894f53229be48fd47bac16e2e22de347a23ae20c71712c38c not found: ID does not exist" Jul 24 12:55:42.294771 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.294702 2554 scope.go:117] "RemoveContainer" containerID="5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4" Jul 24 12:55:42.294893 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:55:42.294878 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4\": container with ID starting with 5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4 not found: ID does not exist" containerID="5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4" Jul 24 12:55:42.294931 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.294897 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4"} err="failed to get container status \"5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4\": rpc error: code = NotFound desc = could not find container \"5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4\": container with ID starting with 5437ada95cbd9a84d21dee0500ae9430048073a005d5ffb131ffe1f5a5a7d7d4 not found: ID does not exist" Jul 24 12:55:42.294931 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.294909 2554 scope.go:117] "RemoveContainer" containerID="2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee" Jul 24 12:55:42.295099 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:55:42.295084 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee\": container with ID starting with 2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee not found: ID does not exist" containerID="2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee" Jul 24 12:55:42.295141 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.295104 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee"} err="failed to get container status \"2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee\": rpc error: code = NotFound desc = could not find container \"2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee\": container with ID starting with 2fd8732f97165fa17b909d0944c07ac74adc792c6ace0103df737c13a81170ee not found: ID does not exist" Jul 24 12:55:42.295141 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.295116 2554 scope.go:117] "RemoveContainer" containerID="773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011" Jul 24 12:55:42.295328 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:55:42.295314 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011\": container with ID starting with 773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011 not found: ID does not exist" containerID="773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011" Jul 24 12:55:42.295367 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.295331 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011"} err="failed to get container status \"773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011\": rpc error: code = NotFound desc = could not find container \"773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011\": container with ID starting with 773bc073b3601149216cb5f4ebe11dcaddbeb97e0bbd480c6119b3fec8aea011 not found: ID does not exist" Jul 24 12:55:42.295367 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.295343 2554 scope.go:117] "RemoveContainer" containerID="ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c" Jul 24 12:55:42.295526 ip-10-0-134-100 kubenswrapper[2554]: E0724 12:55:42.295511 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c\": container with ID starting with ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c not found: ID does not exist" containerID="ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c" Jul 24 12:55:42.295581 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.295529 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c"} err="failed to get container status \"ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c\": rpc error: code = NotFound desc = could not find container \"ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c\": container with ID starting with ca181909664be62b5843fc5a72e10b6e34869f282040800c5708ee5e5504679c not found: ID does not exist" Jul 24 12:55:42.321086 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321058 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 12:55:42.321306 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321295 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy-web" Jul 24 12:55:42.321357 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321308 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy-web" Jul 24 12:55:42.321357 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321327 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="prometheus" Jul 24 12:55:42.321357 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321332 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="prometheus" Jul 24 12:55:42.321357 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321339 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="config-reloader" Jul 24 12:55:42.321357 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321345 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="config-reloader" Jul 24 12:55:42.321357 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321351 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy" Jul 24 12:55:42.321357 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321355 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321361 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="thanos-sidecar" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321366 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="thanos-sidecar" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321371 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy-thanos" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321376 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy-thanos" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321383 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="init-config-reloader" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321388 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="init-config-reloader" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321426 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321435 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy-thanos" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321440 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="config-reloader" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321446 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="kube-rbac-proxy-web" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321451 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="prometheus" Jul 24 12:55:42.321547 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.321455 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="60344f41-064c-4bba-b20f-2695d092c46f" containerName="thanos-sidecar" Jul 24 12:55:42.326708 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.326692 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.331032 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.331014 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 12:55:42.331383 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.331360 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 12:55:42.331383 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.331372 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 12:55:42.331506 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.331490 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4c9hfqbcbm23r\"" Jul 24 12:55:42.332033 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332016 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 12:55:42.332363 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332343 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 12:55:42.332655 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332638 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 12:55:42.332743 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332653 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 12:55:42.332743 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332671 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 12:55:42.332743 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332641 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j5kfz\"" Jul 24 12:55:42.332743 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332703 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 12:55:42.332941 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332854 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 12:55:42.332941 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332859 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 12:55:42.332941 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.332893 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 12:55:42.333191 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.333169 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 12:55:42.341853 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.341832 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 12:55:42.344330 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.344314 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 12:55:42.350797 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.350775 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 12:55:42.439775 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.439737 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.439960 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.439780 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.439960 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.439808 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5586\" (UniqueName: \"kubernetes.io/projected/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-kube-api-access-s5586\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.439960 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.439868 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-config-out\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.439960 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.439896 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-config\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.439960 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.439915 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440166 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.439961 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440166 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.439988 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440166 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440006 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440166 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440030 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440166 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440045 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440166 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440070 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440166 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440094 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440166 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440116 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440166 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440137 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-web-config\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440493 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440172 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440493 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440200 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440493 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440216 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440493 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440249 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.440493 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.440298 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.541160 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541129 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.541301 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541167 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-web-config\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.541301 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541185 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.541301 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541206 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.541301 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541233 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.541301 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541257 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.541301 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541282 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.541630 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541307 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.541630 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541333 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542296 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.541901 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542296 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542196 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542296 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542253 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5586\" (UniqueName: \"kubernetes.io/projected/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-kube-api-access-s5586\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542296 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542290 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-config-out\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542315 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-config\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542337 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542373 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542403 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542433 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542473 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542496 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542549 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542526 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.542959 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.542581 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.544441 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.544414 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.544533 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.544455 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.544623 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.544540 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-web-config\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.544623 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.544608 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.544790 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.544768 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.544983 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.544906 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.545283 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.545260 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.545631 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.545604 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.545851 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.545824 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.546008 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.545981 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.546432 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.546405 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.546520 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.546466 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.546784 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.546761 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.546854 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.546770 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-config-out\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.547083 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.547069 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-config\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.547303 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.547277 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.547599 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.547580 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.555548 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.555528 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5586\" (UniqueName: \"kubernetes.io/projected/af80fbc3-866f-4db1-bfa4-dfba42a6fc22-kube-api-access-s5586\") pod \"prometheus-k8s-0\" (UID: \"af80fbc3-866f-4db1-bfa4-dfba42a6fc22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.596859 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.596827 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60344f41-064c-4bba-b20f-2695d092c46f" path="/var/lib/kubelet/pods/60344f41-064c-4bba-b20f-2695d092c46f/volumes" Jul 24 12:55:42.636250 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.636216 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:42.786188 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:42.786172 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 12:55:42.788467 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:55:42.788435 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf80fbc3_866f_4db1_bfa4_dfba42a6fc22.slice/crio-6fc0085d57fa78992c29520c3091830662abfb26a83bdedeae438831c58f1092 WatchSource:0}: Error finding container 6fc0085d57fa78992c29520c3091830662abfb26a83bdedeae438831c58f1092: Status 404 returned error can't find the container with id 6fc0085d57fa78992c29520c3091830662abfb26a83bdedeae438831c58f1092 Jul 24 12:55:43.248977 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:43.248890 2554 generic.go:356] "Generic (PLEG): container finished" podID="af80fbc3-866f-4db1-bfa4-dfba42a6fc22" containerID="f00c1c272346aa8f09f8ed9714f2c64880488cba8984adc5a49a67956163221a" exitCode=0 Jul 24 12:55:43.248977 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:43.248935 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af80fbc3-866f-4db1-bfa4-dfba42a6fc22","Type":"ContainerDied","Data":"f00c1c272346aa8f09f8ed9714f2c64880488cba8984adc5a49a67956163221a"} Jul 24 12:55:43.249150 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:43.248987 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af80fbc3-866f-4db1-bfa4-dfba42a6fc22","Type":"ContainerStarted","Data":"6fc0085d57fa78992c29520c3091830662abfb26a83bdedeae438831c58f1092"} Jul 24 12:55:43.691479 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:43.691446 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:43.789538 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:43.789516 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:44.255899 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:44.255821 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af80fbc3-866f-4db1-bfa4-dfba42a6fc22","Type":"ContainerStarted","Data":"bf14013e88051446c5f1237de2d53c473ed58e36fbe0eda10986fd1163ac0c70"} Jul 24 12:55:44.256025 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:44.255908 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af80fbc3-866f-4db1-bfa4-dfba42a6fc22","Type":"ContainerStarted","Data":"5a6c3606969283f43ac2bc2c318f5d2e76720b35dcae212d6b2bd108edd52687"} Jul 24 12:55:44.256025 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:44.255936 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af80fbc3-866f-4db1-bfa4-dfba42a6fc22","Type":"ContainerStarted","Data":"f8dfe1154886fa2850a3886a6aad4d59ac5c2978d2a2eba83991d726b5b51121"} Jul 24 12:55:44.256025 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:44.255959 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af80fbc3-866f-4db1-bfa4-dfba42a6fc22","Type":"ContainerStarted","Data":"cabf1a3b8a082c895d884e7aa169827914888bc2faa7724dd0926a5825ed2aee"} Jul 24 12:55:44.256025 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:44.255985 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af80fbc3-866f-4db1-bfa4-dfba42a6fc22","Type":"ContainerStarted","Data":"2a1d7ae4c5562c291396e1ab0500fe627621b009b4043da297a991a6828c99f2"} Jul 24 12:55:44.256025 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:44.256009 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af80fbc3-866f-4db1-bfa4-dfba42a6fc22","Type":"ContainerStarted","Data":"af64aed95057dc06809225b1449095b1d9841d6f97664fadb06caae820d234c9"} Jul 24 12:55:44.302117 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:44.302055 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.2808640860000002 podStartE2EDuration="2.302033765s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.169679ms" isStatefulPod=false podCreationTimestamp="2026-07-24 12:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:55:44.299231352 +0000 UTC m=+160.302533476" watchObservedRunningTime="2026-07-24 12:55:44.302033765 +0000 UTC m=+160.305335870" Jul 24 12:55:47.636596 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:47.636534 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:55:49.600160 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:49.600128 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:49.702921 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:49.702891 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:49.776689 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:49.776658 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:55.194509 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:55.194405 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:55.295267 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:55.295220 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:55:55.363992 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:55:55.363963 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:56:37.410864 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:56:37.410829 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:56:37.511633 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:56:37.511588 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:56:42.636475 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:56:42.636429 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:56:42.652102 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:56:42.652079 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:56:43.470392 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:56:43.470363 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 12:57:06.630314 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:06.630269 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:06.731706 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:06.731670 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:26.308077 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:26.307994 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:28.309423 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:28.309384 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:29.101870 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:29.101836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:32.329840 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:32.329811 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:32.425959 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:32.425932 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:34.153555 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:34.153517 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:34.259266 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:34.259237 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:34.479216 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:34.479130 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:34.577705 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:34.577671 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:36.385290 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:36.385253 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:36.560664 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:36.560628 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:39.130265 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:39.130232 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:39.229742 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:39.229709 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:44.181875 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:44.181840 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:44.602577 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:44.602529 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:44.775006 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:44.774964 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:46.247871 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:46.247841 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:46.348782 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:46.348754 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:50.532813 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:50.532782 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:50.589904 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:50.589873 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:51.626950 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:51.626922 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:53.520738 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:53.520707 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:57:53.757999 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:57:53.757962 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:04.405788 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:04.405749 2554 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 12:58:09.576842 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:09.576801 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:09.680984 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:09.680908 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:10.217305 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.217273 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:10.605292 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.605257 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:10.710223 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.710187 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:10.791713 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.791681 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7"] Jul 24 12:58:10.795281 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.795248 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.799678 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.799643 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 12:58:10.801814 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.801773 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 12:58:10.801960 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.801945 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 12:58:10.802032 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.802019 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pmmpb\"" Jul 24 12:58:10.803608 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.803346 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 12:58:10.814939 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.814914 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7"] Jul 24 12:58:10.888776 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.888696 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df987390-5a29-449a-b657-02ee9407f4ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.888776 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.888740 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df987390-5a29-449a-b657-02ee9407f4ab-tmp\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.888776 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.888768 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jp9\" (UniqueName: \"kubernetes.io/projected/df987390-5a29-449a-b657-02ee9407f4ab-kube-api-access-t7jp9\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.888997 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.888834 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df987390-5a29-449a-b657-02ee9407f4ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.888997 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.888882 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df987390-5a29-449a-b657-02ee9407f4ab-manifests\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.931905 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.931875 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:10.990122 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.990090 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df987390-5a29-449a-b657-02ee9407f4ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.990122 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.990130 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df987390-5a29-449a-b657-02ee9407f4ab-tmp\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.990353 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.990149 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jp9\" (UniqueName: \"kubernetes.io/projected/df987390-5a29-449a-b657-02ee9407f4ab-kube-api-access-t7jp9\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.990394 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.990350 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df987390-5a29-449a-b657-02ee9407f4ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.990437 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.990415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df987390-5a29-449a-b657-02ee9407f4ab-manifests\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.990752 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.990725 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df987390-5a29-449a-b657-02ee9407f4ab-manifests\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.992703 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.992669 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df987390-5a29-449a-b657-02ee9407f4ab-tmp\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.992836 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.992817 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df987390-5a29-449a-b657-02ee9407f4ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:10.992892 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:10.992868 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df987390-5a29-449a-b657-02ee9407f4ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:11.005927 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:11.005900 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jp9\" (UniqueName: \"kubernetes.io/projected/df987390-5a29-449a-b657-02ee9407f4ab-kube-api-access-t7jp9\") pod \"opendatahub-operator-controller-manager-77bd76cdcf-w4fk7\" (UID: \"df987390-5a29-449a-b657-02ee9407f4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:11.024744 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:11.024717 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:11.106878 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:11.106843 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:11.126817 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:11.126773 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:11.219090 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:11.219062 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:11.248414 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:11.248383 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7"] Jul 24 12:58:11.251432 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:58:11.251406 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf987390_5a29_449a_b657_02ee9407f4ab.slice/crio-ada75c37f714ec35aa24201892e5c9f71731a57d721e02fd5123ddc9ede8a935 WatchSource:0}: Error finding container ada75c37f714ec35aa24201892e5c9f71731a57d721e02fd5123ddc9ede8a935: Status 404 returned error can't find the container with id ada75c37f714ec35aa24201892e5c9f71731a57d721e02fd5123ddc9ede8a935 Jul 24 12:58:11.253194 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:11.253178 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 12:58:11.297820 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:11.297792 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:11.751520 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:11.751487 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" event={"ID":"df987390-5a29-449a-b657-02ee9407f4ab","Type":"ContainerStarted","Data":"ada75c37f714ec35aa24201892e5c9f71731a57d721e02fd5123ddc9ede8a935"} Jul 24 12:58:14.765850 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:14.765820 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" event={"ID":"df987390-5a29-449a-b657-02ee9407f4ab","Type":"ContainerStarted","Data":"5f01aa6675d8330a86638a816dd288e5116af98ae82d57c76b22cf078e8074eb"} Jul 24 12:58:15.770662 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:15.770575 2554 generic.go:356] "Generic (PLEG): container finished" podID="df987390-5a29-449a-b657-02ee9407f4ab" containerID="5f01aa6675d8330a86638a816dd288e5116af98ae82d57c76b22cf078e8074eb" exitCode=0 Jul 24 12:58:15.770662 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:15.770625 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" event={"ID":"df987390-5a29-449a-b657-02ee9407f4ab","Type":"ContainerDied","Data":"5f01aa6675d8330a86638a816dd288e5116af98ae82d57c76b22cf078e8074eb"} Jul 24 12:58:16.775414 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:16.775379 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" event={"ID":"df987390-5a29-449a-b657-02ee9407f4ab","Type":"ContainerStarted","Data":"8f777ffedad281f0595551e8bee1493b0752d9b4d4666353f9b103f61862c56e"} Jul 24 12:58:16.775825 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:16.775512 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:16.801195 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:16.801137 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" podStartSLOduration=2.286709681 podStartE2EDuration="6.80111427s" totalImagesPullingTime="3.443807036s" totalInitContainerRuntime="1.070597553s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:58:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:58:16.800292155 +0000 UTC m=+312.803594280" watchObservedRunningTime="2026-07-24 12:58:16.80111427 +0000 UTC m=+312.804416377" Jul 24 12:58:21.992428 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:21.992400 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:22.094738 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:22.094707 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:27.782230 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:27.782200 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77bd76cdcf-w4fk7" Jul 24 12:58:30.011494 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:30.011460 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:30.135334 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:30.135306 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:31.211351 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:31.211322 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:31.341265 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:31.341229 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:31.455032 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:31.454970 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:31.567876 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:31.567845 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:31.676858 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:31.676830 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:32.217535 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:32.217472 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:32.392501 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:32.392470 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:38.792851 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.792812 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk"] Jul 24 12:58:38.800875 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.800857 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.804311 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.804289 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 12:58:38.805516 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.805491 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 12:58:38.805516 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.805510 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 12:58:38.805702 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.805510 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 12:58:38.805702 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.805529 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 12:58:38.805702 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.805494 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2pm28\"" Jul 24 12:58:38.815467 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.815449 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk"] Jul 24 12:58:38.882728 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.882697 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqrz\" (UniqueName: \"kubernetes.io/projected/4e7ef589-9e7f-43bd-8296-c75224f5baeb-kube-api-access-rnqrz\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.882912 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.882750 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e7ef589-9e7f-43bd-8296-c75224f5baeb-cert\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.882912 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.882770 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4e7ef589-9e7f-43bd-8296-c75224f5baeb-manager-config\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.882912 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.882811 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e7ef589-9e7f-43bd-8296-c75224f5baeb-metrics-cert\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.983823 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.983786 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e7ef589-9e7f-43bd-8296-c75224f5baeb-cert\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.983823 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.983823 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4e7ef589-9e7f-43bd-8296-c75224f5baeb-manager-config\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.984036 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.983850 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e7ef589-9e7f-43bd-8296-c75224f5baeb-metrics-cert\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.984036 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.983881 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqrz\" (UniqueName: \"kubernetes.io/projected/4e7ef589-9e7f-43bd-8296-c75224f5baeb-kube-api-access-rnqrz\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.984550 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.984522 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4e7ef589-9e7f-43bd-8296-c75224f5baeb-manager-config\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.986315 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.986284 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e7ef589-9e7f-43bd-8296-c75224f5baeb-metrics-cert\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.986419 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.986343 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e7ef589-9e7f-43bd-8296-c75224f5baeb-cert\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:38.999493 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:38.999466 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqrz\" (UniqueName: \"kubernetes.io/projected/4e7ef589-9e7f-43bd-8296-c75224f5baeb-kube-api-access-rnqrz\") pod \"lws-controller-manager-756cc79bf9-vl2jk\" (UID: \"4e7ef589-9e7f-43bd-8296-c75224f5baeb\") " pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:39.110477 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:39.110374 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:39.245478 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:39.245446 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk"] Jul 24 12:58:39.248893 ip-10-0-134-100 kubenswrapper[2554]: W0724 12:58:39.248865 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7ef589_9e7f_43bd_8296_c75224f5baeb.slice/crio-4eb8827b4867b8ed60ce24d8bafb00fe49173237717c0476303eac28ec6bec47 WatchSource:0}: Error finding container 4eb8827b4867b8ed60ce24d8bafb00fe49173237717c0476303eac28ec6bec47: Status 404 returned error can't find the container with id 4eb8827b4867b8ed60ce24d8bafb00fe49173237717c0476303eac28ec6bec47 Jul 24 12:58:39.860062 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:39.860026 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" event={"ID":"4e7ef589-9e7f-43bd-8296-c75224f5baeb","Type":"ContainerStarted","Data":"4eb8827b4867b8ed60ce24d8bafb00fe49173237717c0476303eac28ec6bec47"} Jul 24 12:58:41.566918 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.566883 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:41.576969 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.576937 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq"] Jul 24 12:58:41.580881 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.580844 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.592084 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.592014 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 12:58:41.592205 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.592177 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 12:58:41.593167 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.593148 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k7h99\"" Jul 24 12:58:41.593274 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.593169 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 12:58:41.597510 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.597467 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 12:58:41.599996 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.599968 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8a3e29-cb8c-4ddd-97c8-c782972a6722-tls-certs\") pod \"kube-auth-proxy-7c6cb77c89-cfxxq\" (UID: \"8d8a3e29-cb8c-4ddd-97c8-c782972a6722\") " pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.600102 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.600027 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d8a3e29-cb8c-4ddd-97c8-c782972a6722-tmp\") pod \"kube-auth-proxy-7c6cb77c89-cfxxq\" (UID: \"8d8a3e29-cb8c-4ddd-97c8-c782972a6722\") " pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.600102 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.600073 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bclxl\" (UniqueName: \"kubernetes.io/projected/8d8a3e29-cb8c-4ddd-97c8-c782972a6722-kube-api-access-bclxl\") pod \"kube-auth-proxy-7c6cb77c89-cfxxq\" (UID: \"8d8a3e29-cb8c-4ddd-97c8-c782972a6722\") " pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.608525 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.608490 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq"] Jul 24 12:58:41.678917 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.678884 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:41.700948 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.700923 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bclxl\" (UniqueName: \"kubernetes.io/projected/8d8a3e29-cb8c-4ddd-97c8-c782972a6722-kube-api-access-bclxl\") pod \"kube-auth-proxy-7c6cb77c89-cfxxq\" (UID: \"8d8a3e29-cb8c-4ddd-97c8-c782972a6722\") " pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.701077 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.700965 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8a3e29-cb8c-4ddd-97c8-c782972a6722-tls-certs\") pod \"kube-auth-proxy-7c6cb77c89-cfxxq\" (UID: \"8d8a3e29-cb8c-4ddd-97c8-c782972a6722\") " pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.701163 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.701139 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d8a3e29-cb8c-4ddd-97c8-c782972a6722-tmp\") pod \"kube-auth-proxy-7c6cb77c89-cfxxq\" (UID: \"8d8a3e29-cb8c-4ddd-97c8-c782972a6722\") " pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.703311 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.703282 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d8a3e29-cb8c-4ddd-97c8-c782972a6722-tmp\") pod \"kube-auth-proxy-7c6cb77c89-cfxxq\" (UID: \"8d8a3e29-cb8c-4ddd-97c8-c782972a6722\") " pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.703400 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.703337 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8a3e29-cb8c-4ddd-97c8-c782972a6722-tls-certs\") pod \"kube-auth-proxy-7c6cb77c89-cfxxq\" (UID: \"8d8a3e29-cb8c-4ddd-97c8-c782972a6722\") " pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.727978 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.727949 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bclxl\" (UniqueName: \"kubernetes.io/projected/8d8a3e29-cb8c-4ddd-97c8-c782972a6722-kube-api-access-bclxl\") pod \"kube-auth-proxy-7c6cb77c89-cfxxq\" (UID: \"8d8a3e29-cb8c-4ddd-97c8-c782972a6722\") " pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:41.870254 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.870165 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" event={"ID":"4e7ef589-9e7f-43bd-8296-c75224f5baeb","Type":"ContainerStarted","Data":"6dad208587275c0aeb1bae57e16e38d11ef982b3c9b5ce1c9c6233d5db3f0004"} Jul 24 12:58:41.870402 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.870305 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:58:41.894218 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:41.894190 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" Jul 24 12:58:42.033448 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:42.033397 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" podStartSLOduration=1.58511274 podStartE2EDuration="4.03338147s" totalImagesPullingTime="2.44826873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:58:42.017986538 +0000 UTC m=+338.021288642" watchObservedRunningTime="2026-07-24 12:58:42.03338147 +0000 UTC m=+338.036683574" Jul 24 12:58:42.132631 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:42.129328 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq"] Jul 24 12:58:42.161797 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:42.159119 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:42.251620 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:42.251586 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:42.382078 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:42.382047 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:42.523262 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:42.523184 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:42.875422 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:42.875388 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" event={"ID":"8d8a3e29-cb8c-4ddd-97c8-c782972a6722","Type":"ContainerStarted","Data":"1e4b3b249181ee74020967def42197951fbd93caeac42851509837bd3e8ba3ff"} Jul 24 12:58:43.464614 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:43.464555 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:43.586516 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:43.586480 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:43.680703 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:43.680669 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:44.889599 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:44.889544 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" event={"ID":"8d8a3e29-cb8c-4ddd-97c8-c782972a6722","Type":"ContainerStarted","Data":"d54816113b41ed7950fc189398371bccc5dedabd79c7247445c1c0381889a7c4"} Jul 24 12:58:44.936432 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:44.936381 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c6cb77c89-cfxxq" podStartSLOduration=1.553671387 podStartE2EDuration="3.936366524s" totalImagesPullingTime="2.382695137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:58:44.935432757 +0000 UTC m=+340.938734880" watchObservedRunningTime="2026-07-24 12:58:44.936366524 +0000 UTC m=+340.939668628" Jul 24 12:58:45.644812 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:45.644780 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:45.780871 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:45.780843 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:47.976760 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:47.976726 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:50.135551 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:50.135517 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:50.250536 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:50.250505 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:51.185940 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:51.185909 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:51.285071 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:51.285028 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:58:52.878145 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:58:52.878113 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-756cc79bf9-vl2jk" Jul 24 12:59:07.122834 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:07.122737 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:07.197576 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:07.197527 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:09.038996 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:09.038962 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:09.116261 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:09.116227 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:29.759526 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:29.759492 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:30.583106 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:30.583076 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:31.651487 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:31.651447 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:31.752007 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:31.751977 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:31.889120 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:31.889086 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:32.597956 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:32.597903 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:32.796218 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:32.796184 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:40.570485 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:40.570448 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:40.671937 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:40.671903 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 12:59:40.738117 ip-10-0-134-100 kubenswrapper[2554]: I0724 12:59:40.738081 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:01.641335 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:01.641298 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:05.053401 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:05.053367 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:05.833081 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:05.833046 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:09.139312 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:09.139277 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:09.242382 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:09.242350 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:09.461689 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:09.461589 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:11.129967 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:11.129936 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:12.240494 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:12.240462 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:13.754710 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:13.754683 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:16.730149 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:16.730113 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:18.731323 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:18.731293 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:18.830695 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:18.830663 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:22.446765 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:22.446731 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:22.524530 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:22.524497 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:22.851194 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:22.851150 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:22.954925 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:22.954896 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:25.758276 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:25.758198 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:26.069118 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:26.069081 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:26.271629 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:26.271597 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:26.378461 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:26.378384 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:30.877551 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:30.877514 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:30.955422 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:30.955389 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:36.649807 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.649774 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf"] Jul 24 13:00:36.653050 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.653033 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:36.659404 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.659169 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7djvc\"" Jul 24 13:00:36.659664 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.659643 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 13:00:36.659757 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.659664 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 13:00:36.677944 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.677914 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf"] Jul 24 13:00:36.740542 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.740508 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf"] Jul 24 13:00:36.740849 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:00:36.740823 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-g6cnk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" podUID="e463abf1-23f9-49f0-a50e-af350ba28e70" Jul 24 13:00:36.746463 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.746430 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf"] Jul 24 13:00:36.809286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.809255 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6cnk\" (UniqueName: \"kubernetes.io/projected/e463abf1-23f9-49f0-a50e-af350ba28e70-kube-api-access-g6cnk\") pod \"kuadrant-operator-controller-manager-86878c769f-ncfsf\" (UID: \"e463abf1-23f9-49f0-a50e-af350ba28e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:36.809441 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.809299 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e463abf1-23f9-49f0-a50e-af350ba28e70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ncfsf\" (UID: \"e463abf1-23f9-49f0-a50e-af350ba28e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:36.910640 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.910535 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6cnk\" (UniqueName: \"kubernetes.io/projected/e463abf1-23f9-49f0-a50e-af350ba28e70-kube-api-access-g6cnk\") pod \"kuadrant-operator-controller-manager-86878c769f-ncfsf\" (UID: \"e463abf1-23f9-49f0-a50e-af350ba28e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:36.910640 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.910615 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e463abf1-23f9-49f0-a50e-af350ba28e70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ncfsf\" (UID: \"e463abf1-23f9-49f0-a50e-af350ba28e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:36.910977 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:36.910959 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e463abf1-23f9-49f0-a50e-af350ba28e70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ncfsf\" (UID: \"e463abf1-23f9-49f0-a50e-af350ba28e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:36.914824 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:00:36.914810 2554 projected.go:196] Error preparing data for projected volume kube-api-access-g6cnk for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-134-100.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object Jul 24 13:00:36.914907 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:00:36.914896 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e463abf1-23f9-49f0-a50e-af350ba28e70-kube-api-access-g6cnk podName:e463abf1-23f9-49f0-a50e-af350ba28e70 nodeName:}" failed. No retries permitted until 2026-07-24 13:00:37.414877638 +0000 UTC m=+453.418179721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g6cnk" (UniqueName: "kubernetes.io/projected/e463abf1-23f9-49f0-a50e-af350ba28e70-kube-api-access-g6cnk") pod "kuadrant-operator-controller-manager-86878c769f-ncfsf" (UID: "e463abf1-23f9-49f0-a50e-af350ba28e70") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-134-100.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object Jul 24 13:00:37.304180 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:37.304149 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:37.306932 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:00:37.306881 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-ncfsf\" is forbidden: User \"system:node:ip-10-0-134-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object" podUID="e463abf1-23f9-49f0-a50e-af350ba28e70" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:37.310905 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:37.310886 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:37.314796 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:00:37.314771 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-ncfsf\" is forbidden: User \"system:node:ip-10-0-134-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object" podUID="e463abf1-23f9-49f0-a50e-af350ba28e70" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:37.415982 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:37.415956 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e463abf1-23f9-49f0-a50e-af350ba28e70-extensions-socket-volume\") pod \"e463abf1-23f9-49f0-a50e-af350ba28e70\" (UID: \"e463abf1-23f9-49f0-a50e-af350ba28e70\") " Jul 24 13:00:37.416104 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:37.416092 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6cnk\" (UniqueName: \"kubernetes.io/projected/e463abf1-23f9-49f0-a50e-af350ba28e70-kube-api-access-g6cnk\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:00:37.416212 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:37.416191 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e463abf1-23f9-49f0-a50e-af350ba28e70-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e463abf1-23f9-49f0-a50e-af350ba28e70" (UID: "e463abf1-23f9-49f0-a50e-af350ba28e70"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:00:37.516831 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:37.516800 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e463abf1-23f9-49f0-a50e-af350ba28e70-extensions-socket-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:00:37.606938 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:37.606856 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:37.714027 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:37.713996 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:38.307221 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:38.307190 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:38.309778 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:00:38.309746 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-ncfsf\" is forbidden: User \"system:node:ip-10-0-134-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object" podUID="e463abf1-23f9-49f0-a50e-af350ba28e70" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:38.315290 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:00:38.315267 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-ncfsf\" is forbidden: User \"system:node:ip-10-0-134-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object" podUID="e463abf1-23f9-49f0-a50e-af350ba28e70" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncfsf" Jul 24 13:00:38.597962 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:38.597886 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e463abf1-23f9-49f0-a50e-af350ba28e70" path="/var/lib/kubelet/pods/e463abf1-23f9-49f0-a50e-af350ba28e70/volumes" Jul 24 13:00:41.362261 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:41.362224 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:41.458859 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:41.458827 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:41.558662 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:41.558632 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:41.618846 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:41.618774 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:45.329932 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:45.329906 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:45.393741 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:45.393713 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:50.947800 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:50.947764 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:51.050669 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:51.050638 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:58.548381 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:58.548351 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:00:58.652257 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:00:58.652226 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:18.503388 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:18.503359 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:18.731118 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:18.731086 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:21.143109 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.143078 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:21.428266 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.428182 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:21.499758 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.499706 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:21.616425 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.616360 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:01:21.619774 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.619757 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:21.622578 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.622544 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 13:01:21.622688 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.622634 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 13:01:21.623655 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.623637 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 13:01:21.623748 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.623654 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nvm6g\"" Jul 24 13:01:21.631617 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.631596 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:01:21.648441 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.648409 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:01:21.775798 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.775709 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxslf\" (UniqueName: \"kubernetes.io/projected/42e606b6-b852-44ba-b52c-b8d77b32eb79-kube-api-access-lxslf\") pod \"limitador-limitador-54d6d988d5-78d5j\" (UID: \"42e606b6-b852-44ba-b52c-b8d77b32eb79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:21.775798 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.775763 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/42e606b6-b852-44ba-b52c-b8d77b32eb79-config-file\") pod \"limitador-limitador-54d6d988d5-78d5j\" (UID: \"42e606b6-b852-44ba-b52c-b8d77b32eb79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:21.876878 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.876827 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxslf\" (UniqueName: \"kubernetes.io/projected/42e606b6-b852-44ba-b52c-b8d77b32eb79-kube-api-access-lxslf\") pod \"limitador-limitador-54d6d988d5-78d5j\" (UID: \"42e606b6-b852-44ba-b52c-b8d77b32eb79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:21.877082 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.876902 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/42e606b6-b852-44ba-b52c-b8d77b32eb79-config-file\") pod \"limitador-limitador-54d6d988d5-78d5j\" (UID: \"42e606b6-b852-44ba-b52c-b8d77b32eb79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:21.877478 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.877459 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/42e606b6-b852-44ba-b52c-b8d77b32eb79-config-file\") pod \"limitador-limitador-54d6d988d5-78d5j\" (UID: \"42e606b6-b852-44ba-b52c-b8d77b32eb79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:21.887198 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.887177 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxslf\" (UniqueName: \"kubernetes.io/projected/42e606b6-b852-44ba-b52c-b8d77b32eb79-kube-api-access-lxslf\") pod \"limitador-limitador-54d6d988d5-78d5j\" (UID: \"42e606b6-b852-44ba-b52c-b8d77b32eb79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:21.930180 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:21.930144 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:22.062322 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.062140 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:01:22.065155 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:01:22.065129 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e606b6_b852_44ba_b52c_b8d77b32eb79.slice/crio-93d7bcb034d918193c951925f95581d8a655e21f7d589a481734f711740adb9a WatchSource:0}: Error finding container 93d7bcb034d918193c951925f95581d8a655e21f7d589a481734f711740adb9a: Status 404 returned error can't find the container with id 93d7bcb034d918193c951925f95581d8a655e21f7d589a481734f711740adb9a Jul 24 13:01:22.194577 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.194544 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:22.270382 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.270347 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-267ph"] Jul 24 13:01:22.272749 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.272727 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:22.275100 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.275087 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-267ph" Jul 24 13:01:22.277801 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.277785 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nbhrf\"" Jul 24 13:01:22.282164 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.281985 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-267ph"] Jul 24 13:01:22.381960 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.381883 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7mv\" (UniqueName: \"kubernetes.io/projected/50fbab45-6f42-4f28-8e72-584f500956cf-kube-api-access-bn7mv\") pod \"authorino-75d556bd77-267ph\" (UID: \"50fbab45-6f42-4f28-8e72-584f500956cf\") " pod="kuadrant-system/authorino-75d556bd77-267ph" Jul 24 13:01:22.463452 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.463422 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" event={"ID":"42e606b6-b852-44ba-b52c-b8d77b32eb79","Type":"ContainerStarted","Data":"93d7bcb034d918193c951925f95581d8a655e21f7d589a481734f711740adb9a"} Jul 24 13:01:22.482907 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.482882 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7mv\" (UniqueName: \"kubernetes.io/projected/50fbab45-6f42-4f28-8e72-584f500956cf-kube-api-access-bn7mv\") pod \"authorino-75d556bd77-267ph\" (UID: \"50fbab45-6f42-4f28-8e72-584f500956cf\") " pod="kuadrant-system/authorino-75d556bd77-267ph" Jul 24 13:01:22.493546 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.493516 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7mv\" (UniqueName: \"kubernetes.io/projected/50fbab45-6f42-4f28-8e72-584f500956cf-kube-api-access-bn7mv\") pod \"authorino-75d556bd77-267ph\" (UID: \"50fbab45-6f42-4f28-8e72-584f500956cf\") " pod="kuadrant-system/authorino-75d556bd77-267ph" Jul 24 13:01:22.584596 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.584545 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-267ph" Jul 24 13:01:22.711694 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:22.711661 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-267ph"] Jul 24 13:01:22.715662 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:01:22.715630 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50fbab45_6f42_4f28_8e72_584f500956cf.slice/crio-2dbd6219f9ea4ae37195d7f9483fa12ddf134d98dfa2140aa51e9d76e8ae20f8 WatchSource:0}: Error finding container 2dbd6219f9ea4ae37195d7f9483fa12ddf134d98dfa2140aa51e9d76e8ae20f8: Status 404 returned error can't find the container with id 2dbd6219f9ea4ae37195d7f9483fa12ddf134d98dfa2140aa51e9d76e8ae20f8 Jul 24 13:01:23.469997 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:23.469959 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-267ph" event={"ID":"50fbab45-6f42-4f28-8e72-584f500956cf","Type":"ContainerStarted","Data":"2dbd6219f9ea4ae37195d7f9483fa12ddf134d98dfa2140aa51e9d76e8ae20f8"} Jul 24 13:01:27.488592 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:27.488537 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" event={"ID":"42e606b6-b852-44ba-b52c-b8d77b32eb79","Type":"ContainerStarted","Data":"2edb5d004e57944b35e3f6f99971b626978ee5efc005ca2797d4b671e9ca95d6"} Jul 24 13:01:27.489094 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:27.488717 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:27.490043 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:27.490023 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-267ph" event={"ID":"50fbab45-6f42-4f28-8e72-584f500956cf","Type":"ContainerStarted","Data":"9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0"} Jul 24 13:01:27.510338 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:27.510295 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" podStartSLOduration=1.599921996 podStartE2EDuration="6.510281684s" totalImagesPullingTime="4.910359688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:01:27.508431503 +0000 UTC m=+503.511733607" watchObservedRunningTime="2026-07-24 13:01:27.510281684 +0000 UTC m=+503.513583787" Jul 24 13:01:27.526960 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:27.526916 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-267ph" podStartSLOduration=1.2640434360000001 podStartE2EDuration="5.526904599s" totalImagesPullingTime="4.262861163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:01:27.525464887 +0000 UTC m=+503.528766992" watchObservedRunningTime="2026-07-24 13:01:27.526904599 +0000 UTC m=+503.530206703" Jul 24 13:01:33.580525 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:33.580490 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:33.657147 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:33.657113 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:38.496169 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:38.496142 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-78d5j" Jul 24 13:01:42.506430 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:42.506387 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:42.604276 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:42.604240 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:43.075102 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:43.075070 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:53.286361 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:53.286329 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:53.386755 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:53.386721 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:56.523129 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:56.523048 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:56.581247 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:56.581210 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:56.733015 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:56.732983 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:56.851106 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:56.851073 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:01:58.253325 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.253295 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-267ph"] Jul 24 13:01:58.253723 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.253473 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-267ph" podUID="50fbab45-6f42-4f28-8e72-584f500956cf" containerName="authorino" containerID="cri-o://9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0" gracePeriod=30 Jul 24 13:01:58.498472 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.498449 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-267ph" Jul 24 13:01:58.580960 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.580927 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn7mv\" (UniqueName: \"kubernetes.io/projected/50fbab45-6f42-4f28-8e72-584f500956cf-kube-api-access-bn7mv\") pod \"50fbab45-6f42-4f28-8e72-584f500956cf\" (UID: \"50fbab45-6f42-4f28-8e72-584f500956cf\") " Jul 24 13:01:58.583168 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.583145 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fbab45-6f42-4f28-8e72-584f500956cf-kube-api-access-bn7mv" (OuterVolumeSpecName: "kube-api-access-bn7mv") pod "50fbab45-6f42-4f28-8e72-584f500956cf" (UID: "50fbab45-6f42-4f28-8e72-584f500956cf"). InnerVolumeSpecName "kube-api-access-bn7mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:01:58.606664 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.606635 2554 generic.go:356] "Generic (PLEG): container finished" podID="50fbab45-6f42-4f28-8e72-584f500956cf" containerID="9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0" exitCode=0 Jul 24 13:01:58.606762 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.606674 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-267ph" Jul 24 13:01:58.606762 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.606715 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-267ph" event={"ID":"50fbab45-6f42-4f28-8e72-584f500956cf","Type":"ContainerDied","Data":"9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0"} Jul 24 13:01:58.606762 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.606748 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-267ph" event={"ID":"50fbab45-6f42-4f28-8e72-584f500956cf","Type":"ContainerDied","Data":"2dbd6219f9ea4ae37195d7f9483fa12ddf134d98dfa2140aa51e9d76e8ae20f8"} Jul 24 13:01:58.606856 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.606767 2554 scope.go:117] "RemoveContainer" containerID="9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0" Jul 24 13:01:58.617031 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.617012 2554 scope.go:117] "RemoveContainer" containerID="9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0" Jul 24 13:01:58.617336 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:01:58.617310 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0\": container with ID starting with 9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0 not found: ID does not exist" containerID="9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0" Jul 24 13:01:58.617436 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.617342 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0"} err="failed to get container status \"9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0\": rpc error: code = NotFound desc = could not find container \"9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0\": container with ID starting with 9be56681789904eadda8976231480aaf3f47fd6b4b1e65f9c66c96ddeb36fdc0 not found: ID does not exist" Jul 24 13:01:58.627978 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.627955 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-267ph"] Jul 24 13:01:58.632730 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.632707 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-267ph"] Jul 24 13:01:58.681723 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:01:58.681688 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn7mv\" (UniqueName: \"kubernetes.io/projected/50fbab45-6f42-4f28-8e72-584f500956cf-kube-api-access-bn7mv\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:02:00.597921 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:00.597881 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50fbab45-6f42-4f28-8e72-584f500956cf" path="/var/lib/kubelet/pods/50fbab45-6f42-4f28-8e72-584f500956cf/volumes" Jul 24 13:02:03.659764 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:03.659729 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:03.772863 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:03.772825 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:03.872271 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:03.872232 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:03.965382 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:03.965293 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:07.061845 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:07.061809 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:07.158671 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:07.158644 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:38.803971 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:38.803927 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:38.903299 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:38.903269 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:39.368107 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:39.368075 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:40.243970 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:40.243937 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:40.338573 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:40.338542 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:40.455923 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:40.455895 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:40.466550 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:40.466519 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:02:51.661800 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:51.661759 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:51.759143 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:51.759111 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:59.463372 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:59.463341 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:02:59.782708 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:02:59.782632 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:03:09.638297 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:09.638194 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:10.014945 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:10.014865 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:16.313903 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.313864 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl"] Jul 24 13:03:16.314319 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.314164 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fbab45-6f42-4f28-8e72-584f500956cf" containerName="authorino" Jul 24 13:03:16.314319 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.314176 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fbab45-6f42-4f28-8e72-584f500956cf" containerName="authorino" Jul 24 13:03:16.314319 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.314229 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fbab45-6f42-4f28-8e72-584f500956cf" containerName="authorino" Jul 24 13:03:16.317034 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.317020 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.320252 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.320226 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 13:03:16.320385 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.320362 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 13:03:16.321446 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.321431 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mpstm\"" Jul 24 13:03:16.321514 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.321461 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 13:03:16.326109 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.326084 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl"] Jul 24 13:03:16.345044 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.345021 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:16.418286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.418256 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.418286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.418290 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61e409e-839e-44a5-a0fc-558f5b674c14-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.418485 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.418322 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.418485 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.418375 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czvst\" (UniqueName: \"kubernetes.io/projected/c61e409e-839e-44a5-a0fc-558f5b674c14-kube-api-access-czvst\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.418485 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.418424 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.418485 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.418470 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.519555 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.519523 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.519741 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.519587 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.519741 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.519614 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.519741 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.519635 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61e409e-839e-44a5-a0fc-558f5b674c14-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.519741 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.519662 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.519741 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.519680 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czvst\" (UniqueName: \"kubernetes.io/projected/c61e409e-839e-44a5-a0fc-558f5b674c14-kube-api-access-czvst\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.519994 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.519971 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.520052 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.519995 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.520052 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.520037 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.522101 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.522078 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61e409e-839e-44a5-a0fc-558f5b674c14-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.522329 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.522314 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61e409e-839e-44a5-a0fc-558f5b674c14-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.530764 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.530745 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czvst\" (UniqueName: \"kubernetes.io/projected/c61e409e-839e-44a5-a0fc-558f5b674c14-kube-api-access-czvst\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svwvl\" (UID: \"c61e409e-839e-44a5-a0fc-558f5b674c14\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.627981 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.627887 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:16.759151 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.759126 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl"] Jul 24 13:03:16.761402 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:03:16.761381 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61e409e_839e_44a5_a0fc_558f5b674c14.slice/crio-899ebbab2c432cfd34956c0f6e1f199acef549927cf1c30c43a54708ac7a99a7 WatchSource:0}: Error finding container 899ebbab2c432cfd34956c0f6e1f199acef549927cf1c30c43a54708ac7a99a7: Status 404 returned error can't find the container with id 899ebbab2c432cfd34956c0f6e1f199acef549927cf1c30c43a54708ac7a99a7 Jul 24 13:03:16.762645 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.762631 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 13:03:16.893971 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:16.893887 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" event={"ID":"c61e409e-839e-44a5-a0fc-558f5b674c14","Type":"ContainerStarted","Data":"899ebbab2c432cfd34956c0f6e1f199acef549927cf1c30c43a54708ac7a99a7"} Jul 24 13:03:17.079975 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:17.079945 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:17.269964 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:17.269888 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:03:22.919461 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:22.919418 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" event={"ID":"c61e409e-839e-44a5-a0fc-558f5b674c14","Type":"ContainerStarted","Data":"2ca9949ce38e2edc8e1a667165ae2a60c33cef90d07405c75eb8d93c1c5dfa48"} Jul 24 13:03:22.943196 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:22.943145 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" podStartSLOduration=1.45631848 podStartE2EDuration="6.943126602s" totalImagesPullingTime="5.486808122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:03:22.941473864 +0000 UTC m=+618.944775970" watchObservedRunningTime="2026-07-24 13:03:22.943126602 +0000 UTC m=+618.946428707" Jul 24 13:03:25.150776 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:25.150684 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:26.165596 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:26.164808 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:03:26.629109 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:26.629073 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:26.629284 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:26.629140 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:26.641676 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:26.641653 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:26.945117 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:26.945034 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svwvl" Jul 24 13:03:29.606366 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:29.606339 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:34.498771 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:34.498733 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:35.778171 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:35.778142 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:03:39.029008 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:39.028974 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:39.131227 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:39.131196 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:39.963462 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:39.963429 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:42.556336 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:42.556300 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:03:44.613215 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.613185 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6"] Jul 24 13:03:44.616337 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.616320 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.619361 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.619345 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 24 13:03:44.631972 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.631950 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6"] Jul 24 13:03:44.643915 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.643893 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zkk\" (UniqueName: \"kubernetes.io/projected/d812304d-423c-4696-bbc4-0b9092529f9d-kube-api-access-p9zkk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.644011 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.643922 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.644011 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.643939 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d812304d-423c-4696-bbc4-0b9092529f9d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.644011 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.643962 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.644182 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.644034 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.644182 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.644086 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.651948 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.651926 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:03:44.745440 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.745406 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zkk\" (UniqueName: \"kubernetes.io/projected/d812304d-423c-4696-bbc4-0b9092529f9d-kube-api-access-p9zkk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.745633 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.745457 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.745633 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.745483 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d812304d-423c-4696-bbc4-0b9092529f9d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.745633 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.745527 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.745633 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.745579 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.745633 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.745627 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.745981 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.745959 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.745981 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.745970 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.746078 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.746059 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.748050 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.748028 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d812304d-423c-4696-bbc4-0b9092529f9d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.748363 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.748343 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d812304d-423c-4696-bbc4-0b9092529f9d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.757918 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.757895 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zkk\" (UniqueName: \"kubernetes.io/projected/d812304d-423c-4696-bbc4-0b9092529f9d-kube-api-access-p9zkk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-74kk6\" (UID: \"d812304d-423c-4696-bbc4-0b9092529f9d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:44.924939 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:44.924846 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:45.080815 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:45.080784 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6"] Jul 24 13:03:45.082687 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:03:45.082661 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd812304d_423c_4696_bbc4_0b9092529f9d.slice/crio-aaeeaa76829d64aa1e136fcf6bb4c78f018136f4255345c1722cb1681a3a2b31 WatchSource:0}: Error finding container aaeeaa76829d64aa1e136fcf6bb4c78f018136f4255345c1722cb1681a3a2b31: Status 404 returned error can't find the container with id aaeeaa76829d64aa1e136fcf6bb4c78f018136f4255345c1722cb1681a3a2b31 Jul 24 13:03:46.008798 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:46.008761 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" event={"ID":"d812304d-423c-4696-bbc4-0b9092529f9d","Type":"ContainerStarted","Data":"d9f64dc7214dc83ea58b3f3b1bc883a2bde99e198cf759cb6fcf045fe0260308"} Jul 24 13:03:46.008798 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:46.008802 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" event={"ID":"d812304d-423c-4696-bbc4-0b9092529f9d","Type":"ContainerStarted","Data":"aaeeaa76829d64aa1e136fcf6bb4c78f018136f4255345c1722cb1681a3a2b31"} Jul 24 13:03:46.030490 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:46.030442 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" podStartSLOduration=1.814686338 podStartE2EDuration="2.030427852s" totalImagesPullingTime="215.741514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:03:46.029180486 +0000 UTC m=+642.032482590" watchObservedRunningTime="2026-07-24 13:03:46.030427852 +0000 UTC m=+642.033729956" Jul 24 13:03:54.671284 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:54.671247 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:03:54.925630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:54.925518 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:54.925781 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:54.925765 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:54.937907 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:54.937878 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:03:55.054069 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:03:55.054038 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-74kk6" Jul 24 13:04:41.793152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.793113 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b84db64b6-2nd7z"] Jul 24 13:04:41.796317 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.796300 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b84db64b6-2nd7z" Jul 24 13:04:41.799724 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.799705 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 24 13:04:41.800870 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.800857 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nbhrf\"" Jul 24 13:04:41.808056 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.808033 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b84db64b6-2nd7z"] Jul 24 13:04:41.831869 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.831834 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4cffa832-4fb9-496a-8a30-28be942b369e-tls-cert\") pod \"authorino-7b84db64b6-2nd7z\" (UID: \"4cffa832-4fb9-496a-8a30-28be942b369e\") " pod="kuadrant-system/authorino-7b84db64b6-2nd7z" Jul 24 13:04:41.832022 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.831927 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbp7s\" (UniqueName: \"kubernetes.io/projected/4cffa832-4fb9-496a-8a30-28be942b369e-kube-api-access-mbp7s\") pod \"authorino-7b84db64b6-2nd7z\" (UID: \"4cffa832-4fb9-496a-8a30-28be942b369e\") " pod="kuadrant-system/authorino-7b84db64b6-2nd7z" Jul 24 13:04:41.933262 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.933226 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbp7s\" (UniqueName: \"kubernetes.io/projected/4cffa832-4fb9-496a-8a30-28be942b369e-kube-api-access-mbp7s\") pod \"authorino-7b84db64b6-2nd7z\" (UID: \"4cffa832-4fb9-496a-8a30-28be942b369e\") " pod="kuadrant-system/authorino-7b84db64b6-2nd7z" Jul 24 13:04:41.933430 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.933279 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4cffa832-4fb9-496a-8a30-28be942b369e-tls-cert\") pod \"authorino-7b84db64b6-2nd7z\" (UID: \"4cffa832-4fb9-496a-8a30-28be942b369e\") " pod="kuadrant-system/authorino-7b84db64b6-2nd7z" Jul 24 13:04:41.935940 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.935919 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4cffa832-4fb9-496a-8a30-28be942b369e-tls-cert\") pod \"authorino-7b84db64b6-2nd7z\" (UID: \"4cffa832-4fb9-496a-8a30-28be942b369e\") " pod="kuadrant-system/authorino-7b84db64b6-2nd7z" Jul 24 13:04:41.942664 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:41.942638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbp7s\" (UniqueName: \"kubernetes.io/projected/4cffa832-4fb9-496a-8a30-28be942b369e-kube-api-access-mbp7s\") pod \"authorino-7b84db64b6-2nd7z\" (UID: \"4cffa832-4fb9-496a-8a30-28be942b369e\") " pod="kuadrant-system/authorino-7b84db64b6-2nd7z" Jul 24 13:04:42.104923 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:42.104848 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b84db64b6-2nd7z" Jul 24 13:04:42.139910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:42.139882 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:04:42.238240 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:42.238217 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b84db64b6-2nd7z"] Jul 24 13:04:42.238540 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:42.238518 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:04:42.240900 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:04:42.240869 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cffa832_4fb9_496a_8a30_28be942b369e.slice/crio-19f415e145b9ba314f4130e7671bbee5533fb63d73bc86217621cd781f29f40c WatchSource:0}: Error finding container 19f415e145b9ba314f4130e7671bbee5533fb63d73bc86217621cd781f29f40c: Status 404 returned error can't find the container with id 19f415e145b9ba314f4130e7671bbee5533fb63d73bc86217621cd781f29f40c Jul 24 13:04:43.224804 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:43.224761 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b84db64b6-2nd7z" event={"ID":"4cffa832-4fb9-496a-8a30-28be942b369e","Type":"ContainerStarted","Data":"3811ddda03ab29972bc0e490e780fff0fd1cfd89164b22e18aa40785764fd95f"} Jul 24 13:04:43.225210 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:43.224813 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b84db64b6-2nd7z" event={"ID":"4cffa832-4fb9-496a-8a30-28be942b369e","Type":"ContainerStarted","Data":"19f415e145b9ba314f4130e7671bbee5533fb63d73bc86217621cd781f29f40c"} Jul 24 13:04:43.248153 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:43.248094 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b84db64b6-2nd7z" podStartSLOduration=1.7372406489999999 podStartE2EDuration="2.24807581s" totalImagesPullingTime="510.835161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:04:43.246493081 +0000 UTC m=+699.249795206" watchObservedRunningTime="2026-07-24 13:04:43.24807581 +0000 UTC m=+699.251377915" Jul 24 13:04:43.342605 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:43.342569 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:04:43.448779 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:04:43.448733 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:05:15.380368 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:05:15.380286 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:05:24.887893 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:05:24.887857 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:05:35.557049 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:05:35.557007 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:05:56.516782 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:05:56.516749 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:06:53.295782 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:06:53.295745 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:06:54.554101 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:06:54.554070 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:07:19.776798 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:19.776760 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:07:28.061438 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:28.061405 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:07:30.327416 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:30.327384 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:07:30.416345 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:30.416316 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:07:30.555199 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:30.555168 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:07:37.005028 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:37.004983 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:07:37.112937 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:37.112906 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:07:37.249121 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:37.249090 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:07:37.372951 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:37.372918 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:07:43.664399 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:43.664368 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:07:43.765608 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:07:43.765577 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:08:01.588785 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:08:01.588705 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:08:11.455267 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:08:11.455234 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:08:21.569630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:08:21.569591 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:08:29.354992 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:08:29.354959 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:08:40.460749 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:08:40.460704 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:08:52.674578 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:08:52.674532 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:09:03.467289 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:09:03.467247 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:09:13.648035 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:09:13.648002 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:09:35.856111 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:09:35.856025 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:09:47.451131 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:09:47.451093 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:09:52.643231 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:09:52.643203 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:10:04.824835 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:10:04.824801 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:10:10.951952 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:10:10.951913 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:10:21.259852 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:10:21.259818 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:10:37.052456 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:10:37.052416 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:10:48.356522 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:10:48.356484 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:10:57.849625 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:10:57.849524 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:11:07.562814 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:11:07.562782 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:11:16.655739 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:11:16.655702 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:12:41.022291 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:12:41.022185 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:13:04.972254 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:13:04.972219 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:13:16.575181 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:13:16.575142 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:13:25.370688 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:13:25.370656 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:13:30.494317 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:13:30.494289 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:13:36.968237 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:13:36.968194 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:13:48.176603 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:13:48.175725 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:13:59.058757 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:13:59.058672 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:14:08.960264 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:14:08.960226 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:14:18.448961 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:14:18.448925 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:14:30.459854 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:14:30.459822 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:14:39.063318 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:14:39.063283 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:14:49.769873 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:14:49.769837 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:15:14.163332 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:15:14.163301 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:15:21.053422 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:15:21.053384 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:15:35.128723 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:15:35.128638 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:15:38.262453 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:15:38.262417 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:15:39.457764 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:15:39.457724 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:15:52.353824 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:15:52.353790 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:16:23.703780 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:16:23.703743 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:16:25.286986 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:16:25.286921 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:16:27.456863 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:16:27.456831 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:16:41.255731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:16:41.255692 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:16:50.354043 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:16:50.354001 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:17:01.350209 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:17:01.350117 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:17:12.156482 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:17:12.156439 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:17:21.660971 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:17:21.660930 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:17:30.355996 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:17:30.355960 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:17:41.556522 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:17:41.556483 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:17:51.152495 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:17:51.152459 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:18:01.761053 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:01.761018 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:18:13.553390 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:13.553353 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:18:18.291440 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:18.291407 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:18:22.647827 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:22.647798 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:18:32.060788 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:32.060712 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:18:41.432413 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:41.432375 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:18:41.535975 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:41.535941 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:18:44.166934 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:44.166901 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:18:49.279379 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:49.279348 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:18:54.661609 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:18:54.661575 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:19:06.958748 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:19:06.958714 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:19:19.162943 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:19:19.162907 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:19:32.148742 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:19:32.148699 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:19:35.434807 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:19:35.434774 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:19:41.448197 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:19:41.448156 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:19:59.248279 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:19:59.248203 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:02.594641 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:02.594604 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:02.690904 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:02.690863 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:03.042099 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:03.042069 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:03.666006 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:03.665969 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:20:09.283017 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:09.282983 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:09.342659 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:09.342633 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:09.537189 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:09.537155 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:09.874669 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:09.874581 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:10.221954 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:10.221869 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:12.419165 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:12.419131 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:12.517901 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:12.517868 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:12.701030 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:12.700941 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:13.036919 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:13.036886 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:13.327024 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:13.326947 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:13.426130 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:13.426100 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:20.253348 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:20.253263 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:20:48.840703 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:48.840666 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:50.730309 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:50.730274 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:51.746123 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:51.746084 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:20:53.237315 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:53.237279 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:53.404843 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:53.404811 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:53.638544 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:53.638514 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:53.938530 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:53.938449 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:54.038993 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:54.038958 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:54.139043 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:54.139009 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:54.457849 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:54.457817 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:54.673380 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:54.673348 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:56.890995 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:56.890961 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:57.009855 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:57.009817 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:57.133872 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:57.133836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:57.547986 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:57.547949 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:20:57.644680 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:20:57.644641 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:03.863086 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:03.863049 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:21:21.856660 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:21.856631 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745"] Jul 24 13:21:21.859720 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:21.859703 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" Jul 24 13:21:21.862176 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:21.862154 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 13:21:21.862280 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:21.862243 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7xjs\"" Jul 24 13:21:21.868694 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:21.868672 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745"] Jul 24 13:21:21.974252 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:21.974214 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a410949-a2f5-47af-81d1-2c400bc07994-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745\" (UID: \"6a410949-a2f5-47af-81d1-2c400bc07994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" Jul 24 13:21:22.075108 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:22.075080 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a410949-a2f5-47af-81d1-2c400bc07994-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745\" (UID: \"6a410949-a2f5-47af-81d1-2c400bc07994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" Jul 24 13:21:22.075675 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:22.075657 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a410949-a2f5-47af-81d1-2c400bc07994-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745\" (UID: \"6a410949-a2f5-47af-81d1-2c400bc07994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" Jul 24 13:21:22.170156 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:22.170080 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" Jul 24 13:21:22.302704 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:22.302673 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745"] Jul 24 13:21:22.305832 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:21:22.305798 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a410949_a2f5_47af_81d1_2c400bc07994.slice/crio-7d7c780be4dcc0274992ca5180aa3ac82bfc838ea9e68004bb77b261eb49db4a WatchSource:0}: Error finding container 7d7c780be4dcc0274992ca5180aa3ac82bfc838ea9e68004bb77b261eb49db4a: Status 404 returned error can't find the container with id 7d7c780be4dcc0274992ca5180aa3ac82bfc838ea9e68004bb77b261eb49db4a Jul 24 13:21:22.307685 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:22.307666 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 13:21:22.315834 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:22.315800 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" event={"ID":"6a410949-a2f5-47af-81d1-2c400bc07994","Type":"ContainerStarted","Data":"7d7c780be4dcc0274992ca5180aa3ac82bfc838ea9e68004bb77b261eb49db4a"} Jul 24 13:21:25.334223 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:25.334187 2554 generic.go:356] "Generic (PLEG): container finished" podID="6a410949-a2f5-47af-81d1-2c400bc07994" containerID="1f02256542ff12faf53669c19527dc06a95c9367711769df80e00706455feba4" exitCode=0 Jul 24 13:21:25.334660 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:25.334245 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" event={"ID":"6a410949-a2f5-47af-81d1-2c400bc07994","Type":"ContainerDied","Data":"1f02256542ff12faf53669c19527dc06a95c9367711769df80e00706455feba4"} Jul 24 13:21:26.462874 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:26.462850 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" Jul 24 13:21:26.620916 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:26.620827 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a410949-a2f5-47af-81d1-2c400bc07994-maas-api-service-ca\") pod \"6a410949-a2f5-47af-81d1-2c400bc07994\" (UID: \"6a410949-a2f5-47af-81d1-2c400bc07994\") " Jul 24 13:21:26.621179 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:26.621155 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a410949-a2f5-47af-81d1-2c400bc07994-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a410949-a2f5-47af-81d1-2c400bc07994" (UID: "6a410949-a2f5-47af-81d1-2c400bc07994"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:21:26.721914 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:26.721879 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a410949-a2f5-47af-81d1-2c400bc07994-maas-api-service-ca\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:21:27.343977 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:27.343951 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" Jul 24 13:21:27.343977 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:27.343963 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745" event={"ID":"6a410949-a2f5-47af-81d1-2c400bc07994","Type":"ContainerDied","Data":"7d7c780be4dcc0274992ca5180aa3ac82bfc838ea9e68004bb77b261eb49db4a"} Jul 24 13:21:27.344181 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:27.344005 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d7c780be4dcc0274992ca5180aa3ac82bfc838ea9e68004bb77b261eb49db4a" Jul 24 13:21:32.493999 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:32.493961 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:32.580752 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:32.580722 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:32.681805 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:32.681774 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:35.063072 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:35.063036 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:21:36.958823 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:36.958777 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:36.970864 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:36.970830 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745"] Jul 24 13:21:36.972914 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:36.972892 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90d0d94f-k2745"] Jul 24 13:21:37.135287 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:37.135251 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:37.361098 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:37.361060 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:37.591732 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:37.591691 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:37.909503 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:37.909473 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:37.987719 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:37.987690 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:38.598549 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:38.598513 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a410949-a2f5-47af-81d1-2c400bc07994" path="/var/lib/kubelet/pods/6a410949-a2f5-47af-81d1-2c400bc07994/volumes" Jul 24 13:21:40.006517 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:40.006467 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:40.121211 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:40.121178 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:40.389262 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:40.389230 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:40.741351 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:40.741266 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:41.230446 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.230416 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:41.352952 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.352918 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:21:41.503968 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.503875 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws"] Jul 24 13:21:41.504256 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.504239 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a410949-a2f5-47af-81d1-2c400bc07994" containerName="revoke-keys" Jul 24 13:21:41.504335 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.504259 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a410949-a2f5-47af-81d1-2c400bc07994" containerName="revoke-keys" Jul 24 13:21:41.504335 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.504329 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a410949-a2f5-47af-81d1-2c400bc07994" containerName="revoke-keys" Jul 24 13:21:41.507359 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.507340 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:21:41.510911 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.510877 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a54eb296\"" Jul 24 13:21:41.511234 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.511207 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a54eb296-dockercfg-764jv\"" Jul 24 13:21:41.521713 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.521695 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws"] Jul 24 13:21:41.536215 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.536194 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cq57\" (UniqueName: \"kubernetes.io/projected/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-kube-api-access-2cq57\") pod \"payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws\" (UID: \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\") " pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:21:41.536312 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.536244 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-plugins-config-volume\") pod \"payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws\" (UID: \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\") " pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:21:41.637378 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.637339 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cq57\" (UniqueName: \"kubernetes.io/projected/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-kube-api-access-2cq57\") pod \"payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws\" (UID: \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\") " pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:21:41.637532 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.637409 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-plugins-config-volume\") pod \"payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws\" (UID: \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\") " pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:21:41.638180 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.638159 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-plugins-config-volume\") pod \"payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws\" (UID: \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\") " pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:21:41.650327 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.650297 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cq57\" (UniqueName: \"kubernetes.io/projected/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-kube-api-access-2cq57\") pod \"payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws\" (UID: \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\") " pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:21:41.817246 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.817219 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:21:41.956284 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:41.956263 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws"] Jul 24 13:21:41.958736 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:21:41.958710 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59d8f4e_7fa6_43a8_9b2a_4fd92c6eb195.slice/crio-6737dce18130774570a3915aefbdf5a7bb3bb8606027725b9574388a23ae9abc WatchSource:0}: Error finding container 6737dce18130774570a3915aefbdf5a7bb3bb8606027725b9574388a23ae9abc: Status 404 returned error can't find the container with id 6737dce18130774570a3915aefbdf5a7bb3bb8606027725b9574388a23ae9abc Jul 24 13:21:42.406757 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:42.406719 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" event={"ID":"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195","Type":"ContainerStarted","Data":"6737dce18130774570a3915aefbdf5a7bb3bb8606027725b9574388a23ae9abc"} Jul 24 13:21:44.416816 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:44.416709 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" event={"ID":"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195","Type":"ContainerStarted","Data":"57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6"} Jul 24 13:21:44.416816 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:44.416792 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:21:44.438528 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:44.438487 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" podStartSLOduration=1.3169370759999999 podStartE2EDuration="3.438472516s" totalImagesPullingTime="2.12153544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:21:44.438003218 +0000 UTC m=+1720.441305322" watchObservedRunningTime="2026-07-24 13:21:44.438472516 +0000 UTC m=+1720.441774620" Jul 24 13:21:47.576463 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:47.576432 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:21:55.422747 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:21:55.422715 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:22:16.520069 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.520037 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:16.573202 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.573166 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws"] Jul 24 13:22:16.573500 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.573461 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" podUID="d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195" containerName="payload-processing" containerID="cri-o://57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6" gracePeriod=30 Jul 24 13:22:16.625482 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.625454 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:16.756036 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.756009 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:16.825725 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.825701 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:22:16.937117 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.937086 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-plugins-config-volume\") pod \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\" (UID: \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\") " Jul 24 13:22:16.937295 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.937157 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cq57\" (UniqueName: \"kubernetes.io/projected/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-kube-api-access-2cq57\") pod \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\" (UID: \"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195\") " Jul 24 13:22:16.937519 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.937496 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195" (UID: "d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:22:16.939511 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:16.939490 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-kube-api-access-2cq57" (OuterVolumeSpecName: "kube-api-access-2cq57") pod "d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195" (UID: "d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195"). InnerVolumeSpecName "kube-api-access-2cq57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:22:17.038144 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.038109 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:22:17.038144 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.038138 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cq57\" (UniqueName: \"kubernetes.io/projected/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195-kube-api-access-2cq57\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:22:17.555997 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.555965 2554 generic.go:356] "Generic (PLEG): container finished" podID="d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195" containerID="57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6" exitCode=0 Jul 24 13:22:17.556414 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.556010 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" Jul 24 13:22:17.556414 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.556019 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" event={"ID":"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195","Type":"ContainerDied","Data":"57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6"} Jul 24 13:22:17.556414 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.556061 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws" event={"ID":"d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195","Type":"ContainerDied","Data":"6737dce18130774570a3915aefbdf5a7bb3bb8606027725b9574388a23ae9abc"} Jul 24 13:22:17.556414 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.556083 2554 scope.go:117] "RemoveContainer" containerID="57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6" Jul 24 13:22:17.565387 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.565364 2554 scope.go:117] "RemoveContainer" containerID="57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6" Jul 24 13:22:17.565661 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:22:17.565636 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6\": container with ID starting with 57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6 not found: ID does not exist" containerID="57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6" Jul 24 13:22:17.565735 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.565676 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6"} err="failed to get container status \"57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6\": rpc error: code = NotFound desc = could not find container \"57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6\": container with ID starting with 57bc3e5fdde1811b0bf1506ee3a5ed53f8bdd869e882ea5a5bbbf2c58f046dc6 not found: ID does not exist" Jul 24 13:22:17.586373 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.586347 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws"] Jul 24 13:22:17.591318 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:17.591292 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a54eb296-f75bb8f6-qlhws"] Jul 24 13:22:18.597315 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:18.597283 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195" path="/var/lib/kubelet/pods/d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195/volumes" Jul 24 13:22:19.474686 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:19.474656 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:22:20.975381 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:20.975346 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:22.229216 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:22.229184 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:22.456927 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:22.456894 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:22.694413 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:22.694371 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:23.019962 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:23.019887 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:23.411365 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:23.411336 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:25.297767 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:25.297724 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:25.397163 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:25.397133 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:25.510767 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:25.510734 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:25.744010 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:25.743922 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:26.251252 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.251219 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:26.367704 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.367677 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:22:26.460448 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.460412 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx"] Jul 24 13:22:26.460728 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.460716 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195" containerName="payload-processing" Jul 24 13:22:26.460776 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.460729 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195" containerName="payload-processing" Jul 24 13:22:26.460837 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.460777 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d59d8f4e-7fa6-43a8-9b2a-4fd92c6eb195" containerName="payload-processing" Jul 24 13:22:26.463178 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.463153 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:26.466001 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.465986 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-40670996-dockercfg-xpn25\"" Jul 24 13:22:26.466085 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.466023 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-40670996\"" Jul 24 13:22:26.473207 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.473188 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx"] Jul 24 13:22:26.534155 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.534053 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp"] Jul 24 13:22:26.538299 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.538260 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:26.545286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.545258 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp"] Jul 24 13:22:26.625584 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.625521 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9sll\" (UniqueName: \"kubernetes.io/projected/87b90c54-4c79-452d-b6b6-2656f3788dfc-kube-api-access-j9sll\") pod \"payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx\" (UID: \"87b90c54-4c79-452d-b6b6-2656f3788dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:26.625777 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.625657 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b90c54-4c79-452d-b6b6-2656f3788dfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx\" (UID: \"87b90c54-4c79-452d-b6b6-2656f3788dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:26.726763 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.726723 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df3395e-405c-41df-ad5e-b35e018c3cb0-plugins-config-volume\") pod \"payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp\" (UID: \"4df3395e-405c-41df-ad5e-b35e018c3cb0\") " pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:26.726938 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.726799 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgsgv\" (UniqueName: \"kubernetes.io/projected/4df3395e-405c-41df-ad5e-b35e018c3cb0-kube-api-access-dgsgv\") pod \"payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp\" (UID: \"4df3395e-405c-41df-ad5e-b35e018c3cb0\") " pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:26.726938 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.726849 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b90c54-4c79-452d-b6b6-2656f3788dfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx\" (UID: \"87b90c54-4c79-452d-b6b6-2656f3788dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:26.726938 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.726903 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9sll\" (UniqueName: \"kubernetes.io/projected/87b90c54-4c79-452d-b6b6-2656f3788dfc-kube-api-access-j9sll\") pod \"payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx\" (UID: \"87b90c54-4c79-452d-b6b6-2656f3788dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:26.727734 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.727716 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b90c54-4c79-452d-b6b6-2656f3788dfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx\" (UID: \"87b90c54-4c79-452d-b6b6-2656f3788dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:26.741160 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.741128 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9sll\" (UniqueName: \"kubernetes.io/projected/87b90c54-4c79-452d-b6b6-2656f3788dfc-kube-api-access-j9sll\") pod \"payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx\" (UID: \"87b90c54-4c79-452d-b6b6-2656f3788dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:26.773495 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.773456 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:26.828234 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.828207 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df3395e-405c-41df-ad5e-b35e018c3cb0-plugins-config-volume\") pod \"payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp\" (UID: \"4df3395e-405c-41df-ad5e-b35e018c3cb0\") " pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:26.828384 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.828251 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsgv\" (UniqueName: \"kubernetes.io/projected/4df3395e-405c-41df-ad5e-b35e018c3cb0-kube-api-access-dgsgv\") pod \"payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp\" (UID: \"4df3395e-405c-41df-ad5e-b35e018c3cb0\") " pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:26.828901 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.828879 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df3395e-405c-41df-ad5e-b35e018c3cb0-plugins-config-volume\") pod \"payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp\" (UID: \"4df3395e-405c-41df-ad5e-b35e018c3cb0\") " pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:26.839313 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.839285 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsgv\" (UniqueName: \"kubernetes.io/projected/4df3395e-405c-41df-ad5e-b35e018c3cb0-kube-api-access-dgsgv\") pod \"payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp\" (UID: \"4df3395e-405c-41df-ad5e-b35e018c3cb0\") " pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:26.850263 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.850240 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:26.909300 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:26.909268 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx"] Jul 24 13:22:26.911658 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:22:26.911616 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b90c54_4c79_452d_b6b6_2656f3788dfc.slice/crio-fd3992495e6c513be4fdbdd57d2cd4c9cc293cb7ab198fbe410c145fa8e51308 WatchSource:0}: Error finding container fd3992495e6c513be4fdbdd57d2cd4c9cc293cb7ab198fbe410c145fa8e51308: Status 404 returned error can't find the container with id fd3992495e6c513be4fdbdd57d2cd4c9cc293cb7ab198fbe410c145fa8e51308 Jul 24 13:22:27.001687 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:27.001519 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp"] Jul 24 13:22:27.003757 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:22:27.003735 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df3395e_405c_41df_ad5e_b35e018c3cb0.slice/crio-3f9321fa98e30f4f7b4195be34c0dcc233969447f9d89944939ceae3bcd9f711 WatchSource:0}: Error finding container 3f9321fa98e30f4f7b4195be34c0dcc233969447f9d89944939ceae3bcd9f711: Status 404 returned error can't find the container with id 3f9321fa98e30f4f7b4195be34c0dcc233969447f9d89944939ceae3bcd9f711 Jul 24 13:22:27.602933 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:27.602900 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" event={"ID":"87b90c54-4c79-452d-b6b6-2656f3788dfc","Type":"ContainerStarted","Data":"a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19"} Jul 24 13:22:27.603377 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:27.602948 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" event={"ID":"87b90c54-4c79-452d-b6b6-2656f3788dfc","Type":"ContainerStarted","Data":"fd3992495e6c513be4fdbdd57d2cd4c9cc293cb7ab198fbe410c145fa8e51308"} Jul 24 13:22:27.603377 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:27.603026 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:27.604414 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:27.604392 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" event={"ID":"4df3395e-405c-41df-ad5e-b35e018c3cb0","Type":"ContainerStarted","Data":"2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a"} Jul 24 13:22:27.604469 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:27.604436 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" event={"ID":"4df3395e-405c-41df-ad5e-b35e018c3cb0","Type":"ContainerStarted","Data":"3f9321fa98e30f4f7b4195be34c0dcc233969447f9d89944939ceae3bcd9f711"} Jul 24 13:22:27.604528 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:27.604514 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:27.621428 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:27.621378 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" podStartSLOduration=1.6213649 podStartE2EDuration="1.6213649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:22:27.619750536 +0000 UTC m=+1763.623052641" watchObservedRunningTime="2026-07-24 13:22:27.6213649 +0000 UTC m=+1763.624667003" Jul 24 13:22:27.638338 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:27.638292 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" podStartSLOduration=1.638274171 podStartE2EDuration="1.638274171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:22:27.637668085 +0000 UTC m=+1763.640970205" watchObservedRunningTime="2026-07-24 13:22:27.638274171 +0000 UTC m=+1763.641576278" Jul 24 13:22:31.358275 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:31.358232 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:22:38.611724 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:38.611691 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:22:38.612081 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:38.611774 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:22:50.145428 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.145392 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2"] Jul 24 13:22:50.147639 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.147625 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" Jul 24 13:22:50.154744 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.154575 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 13:22:50.160710 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.155168 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7xjs\"" Jul 24 13:22:50.162045 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.162014 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2"] Jul 24 13:22:50.205864 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.205832 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d004cf6-a6de-463d-9fba-4acb116ef2c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-40670996-fl6p2\" (UID: \"6d004cf6-a6de-463d-9fba-4acb116ef2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" Jul 24 13:22:50.306486 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.306452 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d004cf6-a6de-463d-9fba-4acb116ef2c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-40670996-fl6p2\" (UID: \"6d004cf6-a6de-463d-9fba-4acb116ef2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" Jul 24 13:22:50.307149 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.307122 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d004cf6-a6de-463d-9fba-4acb116ef2c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-40670996-fl6p2\" (UID: \"6d004cf6-a6de-463d-9fba-4acb116ef2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" Jul 24 13:22:50.463708 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.463623 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" Jul 24 13:22:50.598586 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.598537 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2"] Jul 24 13:22:50.600436 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:22:50.600402 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d004cf6_a6de_463d_9fba_4acb116ef2c4.slice/crio-391de5bd43b780d199d914d0b7b0586a5c4d27f1b067b5f220a93f9377f17485 WatchSource:0}: Error finding container 391de5bd43b780d199d914d0b7b0586a5c4d27f1b067b5f220a93f9377f17485: Status 404 returned error can't find the container with id 391de5bd43b780d199d914d0b7b0586a5c4d27f1b067b5f220a93f9377f17485 Jul 24 13:22:50.707121 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:50.707075 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" event={"ID":"6d004cf6-a6de-463d-9fba-4acb116ef2c4","Type":"ContainerStarted","Data":"391de5bd43b780d199d914d0b7b0586a5c4d27f1b067b5f220a93f9377f17485"} Jul 24 13:22:51.712548 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:51.712450 2554 generic.go:356] "Generic (PLEG): container finished" podID="6d004cf6-a6de-463d-9fba-4acb116ef2c4" containerID="160aad7dd893bfd3be84c3c223109c000ec23544795a5325889112ff6a9dabfb" exitCode=0 Jul 24 13:22:51.712548 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:51.712499 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" event={"ID":"6d004cf6-a6de-463d-9fba-4acb116ef2c4","Type":"ContainerDied","Data":"160aad7dd893bfd3be84c3c223109c000ec23544795a5325889112ff6a9dabfb"} Jul 24 13:22:52.843377 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:52.843352 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" Jul 24 13:22:52.928509 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:52.928470 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d004cf6-a6de-463d-9fba-4acb116ef2c4-maas-api-service-ca\") pod \"6d004cf6-a6de-463d-9fba-4acb116ef2c4\" (UID: \"6d004cf6-a6de-463d-9fba-4acb116ef2c4\") " Jul 24 13:22:52.928911 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:52.928884 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d004cf6-a6de-463d-9fba-4acb116ef2c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d004cf6-a6de-463d-9fba-4acb116ef2c4" (UID: "6d004cf6-a6de-463d-9fba-4acb116ef2c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:22:53.029444 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:53.029348 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d004cf6-a6de-463d-9fba-4acb116ef2c4-maas-api-service-ca\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:22:53.721926 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:53.721892 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" event={"ID":"6d004cf6-a6de-463d-9fba-4acb116ef2c4","Type":"ContainerDied","Data":"391de5bd43b780d199d914d0b7b0586a5c4d27f1b067b5f220a93f9377f17485"} Jul 24 13:22:53.721926 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:53.721913 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2" Jul 24 13:22:53.722130 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:22:53.721938 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="391de5bd43b780d199d914d0b7b0586a5c4d27f1b067b5f220a93f9377f17485" Jul 24 13:23:00.725176 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:00.725085 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:00.768123 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:00.768090 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp"] Jul 24 13:23:00.768356 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:00.768332 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" podUID="4df3395e-405c-41df-ad5e-b35e018c3cb0" containerName="payload-processing" containerID="cri-o://2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a" gracePeriod=30 Jul 24 13:23:00.779679 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:00.779648 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx"] Jul 24 13:23:00.779913 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:00.779868 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" podUID="87b90c54-4c79-452d-b6b6-2656f3788dfc" containerName="payload-pre-processing" containerID="cri-o://a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19" gracePeriod=30 Jul 24 13:23:00.830883 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:00.830856 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:00.931228 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:00.931199 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:01.047144 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.047124 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:23:01.049874 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.049856 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:23:01.100506 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.100479 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9sll\" (UniqueName: \"kubernetes.io/projected/87b90c54-4c79-452d-b6b6-2656f3788dfc-kube-api-access-j9sll\") pod \"87b90c54-4c79-452d-b6b6-2656f3788dfc\" (UID: \"87b90c54-4c79-452d-b6b6-2656f3788dfc\") " Jul 24 13:23:01.100704 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.100572 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b90c54-4c79-452d-b6b6-2656f3788dfc-plugins-config-volume\") pod \"87b90c54-4c79-452d-b6b6-2656f3788dfc\" (UID: \"87b90c54-4c79-452d-b6b6-2656f3788dfc\") " Jul 24 13:23:01.100704 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.100607 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgsgv\" (UniqueName: \"kubernetes.io/projected/4df3395e-405c-41df-ad5e-b35e018c3cb0-kube-api-access-dgsgv\") pod \"4df3395e-405c-41df-ad5e-b35e018c3cb0\" (UID: \"4df3395e-405c-41df-ad5e-b35e018c3cb0\") " Jul 24 13:23:01.100704 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.100627 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df3395e-405c-41df-ad5e-b35e018c3cb0-plugins-config-volume\") pod \"4df3395e-405c-41df-ad5e-b35e018c3cb0\" (UID: \"4df3395e-405c-41df-ad5e-b35e018c3cb0\") " Jul 24 13:23:01.100950 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.100922 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b90c54-4c79-452d-b6b6-2656f3788dfc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87b90c54-4c79-452d-b6b6-2656f3788dfc" (UID: "87b90c54-4c79-452d-b6b6-2656f3788dfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:23:01.101162 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.101135 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df3395e-405c-41df-ad5e-b35e018c3cb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4df3395e-405c-41df-ad5e-b35e018c3cb0" (UID: "4df3395e-405c-41df-ad5e-b35e018c3cb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:23:01.102926 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.102898 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df3395e-405c-41df-ad5e-b35e018c3cb0-kube-api-access-dgsgv" (OuterVolumeSpecName: "kube-api-access-dgsgv") pod "4df3395e-405c-41df-ad5e-b35e018c3cb0" (UID: "4df3395e-405c-41df-ad5e-b35e018c3cb0"). InnerVolumeSpecName "kube-api-access-dgsgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:23:01.103026 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.102997 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b90c54-4c79-452d-b6b6-2656f3788dfc-kube-api-access-j9sll" (OuterVolumeSpecName: "kube-api-access-j9sll") pod "87b90c54-4c79-452d-b6b6-2656f3788dfc" (UID: "87b90c54-4c79-452d-b6b6-2656f3788dfc"). InnerVolumeSpecName "kube-api-access-j9sll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:23:01.202188 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.202147 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b90c54-4c79-452d-b6b6-2656f3788dfc-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:23:01.202188 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.202180 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgsgv\" (UniqueName: \"kubernetes.io/projected/4df3395e-405c-41df-ad5e-b35e018c3cb0-kube-api-access-dgsgv\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:23:01.202188 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.202189 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df3395e-405c-41df-ad5e-b35e018c3cb0-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:23:01.202188 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.202198 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9sll\" (UniqueName: \"kubernetes.io/projected/87b90c54-4c79-452d-b6b6-2656f3788dfc-kube-api-access-j9sll\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:23:01.760776 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.760746 2554 generic.go:356] "Generic (PLEG): container finished" podID="87b90c54-4c79-452d-b6b6-2656f3788dfc" containerID="a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19" exitCode=0 Jul 24 13:23:01.761274 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.760795 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" Jul 24 13:23:01.761274 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.760831 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" event={"ID":"87b90c54-4c79-452d-b6b6-2656f3788dfc","Type":"ContainerDied","Data":"a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19"} Jul 24 13:23:01.761274 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.760893 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx" event={"ID":"87b90c54-4c79-452d-b6b6-2656f3788dfc","Type":"ContainerDied","Data":"fd3992495e6c513be4fdbdd57d2cd4c9cc293cb7ab198fbe410c145fa8e51308"} Jul 24 13:23:01.761274 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.760927 2554 scope.go:117] "RemoveContainer" containerID="a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19" Jul 24 13:23:01.762296 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.762218 2554 generic.go:356] "Generic (PLEG): container finished" podID="4df3395e-405c-41df-ad5e-b35e018c3cb0" containerID="2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a" exitCode=0 Jul 24 13:23:01.762296 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.762265 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" event={"ID":"4df3395e-405c-41df-ad5e-b35e018c3cb0","Type":"ContainerDied","Data":"2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a"} Jul 24 13:23:01.762469 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.762313 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" event={"ID":"4df3395e-405c-41df-ad5e-b35e018c3cb0","Type":"ContainerDied","Data":"3f9321fa98e30f4f7b4195be34c0dcc233969447f9d89944939ceae3bcd9f711"} Jul 24 13:23:01.762469 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.762271 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp" Jul 24 13:23:01.770757 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.770537 2554 scope.go:117] "RemoveContainer" containerID="a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19" Jul 24 13:23:01.771032 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:23:01.770981 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19\": container with ID starting with a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19 not found: ID does not exist" containerID="a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19" Jul 24 13:23:01.771093 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.771025 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19"} err="failed to get container status \"a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19\": rpc error: code = NotFound desc = could not find container \"a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19\": container with ID starting with a0e4fdf6dca9ad8fee3c3b9f2569f0b16fe0c39c9e26027fb6384e6fe400eb19 not found: ID does not exist" Jul 24 13:23:01.771093 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.771048 2554 scope.go:117] "RemoveContainer" containerID="2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a" Jul 24 13:23:01.779112 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.779093 2554 scope.go:117] "RemoveContainer" containerID="2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a" Jul 24 13:23:01.779371 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:23:01.779353 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a\": container with ID starting with 2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a not found: ID does not exist" containerID="2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a" Jul 24 13:23:01.779437 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.779382 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a"} err="failed to get container status \"2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a\": rpc error: code = NotFound desc = could not find container \"2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a\": container with ID starting with 2b3e35a7f08ab93766c51fb7e05f14b514ade3ce7025be0b3c83cadac6ad810a not found: ID does not exist" Jul 24 13:23:01.790251 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.790231 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp"] Jul 24 13:23:01.796110 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.796088 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-40670996-6c9947fdb6-rjchp"] Jul 24 13:23:01.809670 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.809650 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx"] Jul 24 13:23:01.818194 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:01.818167 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-40670996-6786599cb4-t72gx"] Jul 24 13:23:02.598736 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:02.598704 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df3395e-405c-41df-ad5e-b35e018c3cb0" path="/var/lib/kubelet/pods/4df3395e-405c-41df-ad5e-b35e018c3cb0/volumes" Jul 24 13:23:02.599062 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:02.599047 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b90c54-4c79-452d-b6b6-2656f3788dfc" path="/var/lib/kubelet/pods/87b90c54-4c79-452d-b6b6-2656f3788dfc/volumes" Jul 24 13:23:03.969119 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:03.969080 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:23:05.257432 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:05.257402 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:05.279520 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:05.279489 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2"] Jul 24 13:23:05.283966 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:05.283942 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-40670996-fl6p2"] Jul 24 13:23:05.474704 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:05.474670 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:05.720722 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:05.720685 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:06.218133 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:06.218100 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:06.311502 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:06.311474 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:06.441672 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:06.441641 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:06.597874 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:06.597846 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d004cf6-a6de-463d-9fba-4acb116ef2c4" path="/var/lib/kubelet/pods/6d004cf6-a6de-463d-9fba-4acb116ef2c4/volumes" Jul 24 13:23:07.984287 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:07.984254 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:15.651869 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:15.651799 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:15.809461 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:15.809418 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:15.952212 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:15.952116 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:21.545596 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:21.545545 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:47.540469 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:47.540429 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:54.903552 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:54.903497 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:54.983315 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:54.983285 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:55.079478 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:55.079443 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:23:55.277594 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:23:55.277490 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:11.102999 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:11.102963 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:12.432053 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:12.432021 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:12.532871 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:12.532837 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:17.746033 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:17.745996 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:17.846578 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:17.846534 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:17.948040 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:17.948009 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:28.512515 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:28.512424 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:29.335247 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:29.335210 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:37.076683 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:37.076650 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:37.179879 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:37.179848 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:37.278016 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:37.277982 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:37.377132 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:37.377042 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:38.211458 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:38.211424 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:38.312841 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:38.312806 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:38.410603 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:38.410574 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:40.154659 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:40.154626 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:42.282779 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:42.282743 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:24:47.815782 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:47.815747 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:47.915404 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:47.915365 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:48.520476 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:48.520437 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:51.058189 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:51.058157 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:24:55.387299 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:55.387270 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:55.488928 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:55.488889 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:55.679081 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:55.679001 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:56.136906 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:56.136870 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:56.246946 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:56.246916 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:58.872835 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:58.872807 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:58.972581 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:58.972539 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:59.073358 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.073326 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:59.597067 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.597037 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:59.699265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.699233 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:24:59.781957 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.781921 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj"] Jul 24 13:24:59.782214 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.782203 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df3395e-405c-41df-ad5e-b35e018c3cb0" containerName="payload-processing" Jul 24 13:24:59.782257 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.782217 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df3395e-405c-41df-ad5e-b35e018c3cb0" containerName="payload-processing" Jul 24 13:24:59.782257 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.782237 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d004cf6-a6de-463d-9fba-4acb116ef2c4" containerName="revoke-keys" Jul 24 13:24:59.782257 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.782244 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d004cf6-a6de-463d-9fba-4acb116ef2c4" containerName="revoke-keys" Jul 24 13:24:59.782257 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.782251 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b90c54-4c79-452d-b6b6-2656f3788dfc" containerName="payload-pre-processing" Jul 24 13:24:59.782257 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.782256 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b90c54-4c79-452d-b6b6-2656f3788dfc" containerName="payload-pre-processing" Jul 24 13:24:59.782401 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.782300 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b90c54-4c79-452d-b6b6-2656f3788dfc" containerName="payload-pre-processing" Jul 24 13:24:59.782401 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.782309 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d004cf6-a6de-463d-9fba-4acb116ef2c4" containerName="revoke-keys" Jul 24 13:24:59.782401 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.782315 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df3395e-405c-41df-ad5e-b35e018c3cb0" containerName="payload-processing" Jul 24 13:24:59.784677 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.784663 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:24:59.787547 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.787528 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b1c83c54-dockercfg-bg4kx\"" Jul 24 13:24:59.789876 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.789856 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b1c83c54\"" Jul 24 13:24:59.797505 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.797483 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj"] Jul 24 13:24:59.908427 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.908335 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9ml\" (UniqueName: \"kubernetes.io/projected/7e244413-4359-49e4-8415-1e95789ee09f-kube-api-access-cf9ml\") pod \"payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj\" (UID: \"7e244413-4359-49e4-8415-1e95789ee09f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:24:59.908427 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.908413 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e244413-4359-49e4-8415-1e95789ee09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj\" (UID: \"7e244413-4359-49e4-8415-1e95789ee09f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:24:59.910050 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.910022 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x"] Jul 24 13:24:59.912359 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.912340 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:24:59.927007 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:24:59.926985 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x"] Jul 24 13:25:00.009232 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.009187 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9ml\" (UniqueName: \"kubernetes.io/projected/7e244413-4359-49e4-8415-1e95789ee09f-kube-api-access-cf9ml\") pod \"payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj\" (UID: \"7e244413-4359-49e4-8415-1e95789ee09f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:25:00.009232 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.009241 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-plugins-config-volume\") pod \"payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x\" (UID: \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\") " pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:00.009467 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.009268 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e244413-4359-49e4-8415-1e95789ee09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj\" (UID: \"7e244413-4359-49e4-8415-1e95789ee09f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:25:00.009467 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.009370 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxb7\" (UniqueName: \"kubernetes.io/projected/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-kube-api-access-9hxb7\") pod \"payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x\" (UID: \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\") " pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:00.009927 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.009911 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e244413-4359-49e4-8415-1e95789ee09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj\" (UID: \"7e244413-4359-49e4-8415-1e95789ee09f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:25:00.024703 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.024681 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9ml\" (UniqueName: \"kubernetes.io/projected/7e244413-4359-49e4-8415-1e95789ee09f-kube-api-access-cf9ml\") pod \"payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj\" (UID: \"7e244413-4359-49e4-8415-1e95789ee09f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:25:00.094297 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.094263 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:25:00.110702 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.110657 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxb7\" (UniqueName: \"kubernetes.io/projected/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-kube-api-access-9hxb7\") pod \"payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x\" (UID: \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\") " pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:00.110834 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.110785 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-plugins-config-volume\") pod \"payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x\" (UID: \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\") " pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:00.111648 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.111625 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-plugins-config-volume\") pod \"payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x\" (UID: \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\") " pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:00.120356 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.120333 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxb7\" (UniqueName: \"kubernetes.io/projected/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-kube-api-access-9hxb7\") pod \"payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x\" (UID: \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\") " pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:00.223087 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.223055 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:00.235080 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.235054 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj"] Jul 24 13:25:00.236498 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:25:00.236465 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e244413_4359_49e4_8415_1e95789ee09f.slice/crio-6655c86b54482304c7489d64e949bacda1e281a529e2d74a7a44cb6689ea178b WatchSource:0}: Error finding container 6655c86b54482304c7489d64e949bacda1e281a529e2d74a7a44cb6689ea178b: Status 404 returned error can't find the container with id 6655c86b54482304c7489d64e949bacda1e281a529e2d74a7a44cb6689ea178b Jul 24 13:25:00.258086 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.258059 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" event={"ID":"7e244413-4359-49e4-8415-1e95789ee09f","Type":"ContainerStarted","Data":"6655c86b54482304c7489d64e949bacda1e281a529e2d74a7a44cb6689ea178b"} Jul 24 13:25:00.360445 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:00.360345 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x"] Jul 24 13:25:00.362326 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:25:00.362299 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff43e3d_55c9_4b8a_9d6d_9788e03d2be4.slice/crio-59e1b1437032ea5c2660b612510152a4e3e72b51ccc2f851d295f289d322308e WatchSource:0}: Error finding container 59e1b1437032ea5c2660b612510152a4e3e72b51ccc2f851d295f289d322308e: Status 404 returned error can't find the container with id 59e1b1437032ea5c2660b612510152a4e3e72b51ccc2f851d295f289d322308e Jul 24 13:25:01.264757 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:01.264710 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" event={"ID":"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4","Type":"ContainerStarted","Data":"d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c"} Jul 24 13:25:01.265198 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:01.264780 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" event={"ID":"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4","Type":"ContainerStarted","Data":"59e1b1437032ea5c2660b612510152a4e3e72b51ccc2f851d295f289d322308e"} Jul 24 13:25:01.265198 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:01.264807 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:01.266194 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:01.266169 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" event={"ID":"7e244413-4359-49e4-8415-1e95789ee09f","Type":"ContainerStarted","Data":"782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c"} Jul 24 13:25:01.266307 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:01.266297 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:25:01.282684 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:01.282630 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" podStartSLOduration=2.282615619 podStartE2EDuration="2.282615619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:25:01.281884688 +0000 UTC m=+1917.285186793" watchObservedRunningTime="2026-07-24 13:25:01.282615619 +0000 UTC m=+1917.285917701" Jul 24 13:25:01.302304 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:01.302242 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" podStartSLOduration=2.3022259800000002 podStartE2EDuration="2.30222598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:25:01.300958492 +0000 UTC m=+1917.304260622" watchObservedRunningTime="2026-07-24 13:25:01.30222598 +0000 UTC m=+1917.305528084" Jul 24 13:25:01.754361 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:01.754327 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:25:03.353145 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:03.353113 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:04.030304 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:04.030270 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:04.129736 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:04.129689 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:04.462453 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:04.462412 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:10.981137 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:10.981101 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:11.081727 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.081687 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:11.348148 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.348114 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc"] Jul 24 13:25:11.350208 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.350194 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" Jul 24 13:25:11.352805 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.352784 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 13:25:11.352931 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.352871 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7xjs\"" Jul 24 13:25:11.359385 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.359353 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc"] Jul 24 13:25:11.412652 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.412618 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05074f7e-d0b9-459d-8c68-326c2608af88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc\" (UID: \"05074f7e-d0b9-459d-8c68-326c2608af88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" Jul 24 13:25:11.514062 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.514023 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05074f7e-d0b9-459d-8c68-326c2608af88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc\" (UID: \"05074f7e-d0b9-459d-8c68-326c2608af88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" Jul 24 13:25:11.514690 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.514664 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05074f7e-d0b9-459d-8c68-326c2608af88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc\" (UID: \"05074f7e-d0b9-459d-8c68-326c2608af88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" Jul 24 13:25:11.661448 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.661340 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" Jul 24 13:25:11.794587 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:11.794419 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc"] Jul 24 13:25:11.797296 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:25:11.797250 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05074f7e_d0b9_459d_8c68_326c2608af88.slice/crio-3a63201f2583c115170b15619540cb62ba808c452f2e3f70d88d2b44a9effd02 WatchSource:0}: Error finding container 3a63201f2583c115170b15619540cb62ba808c452f2e3f70d88d2b44a9effd02: Status 404 returned error can't find the container with id 3a63201f2583c115170b15619540cb62ba808c452f2e3f70d88d2b44a9effd02 Jul 24 13:25:12.273772 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:12.273738 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:25:12.274172 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:12.273820 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:12.316020 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:12.315988 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" event={"ID":"05074f7e-d0b9-459d-8c68-326c2608af88","Type":"ContainerStarted","Data":"7bdc5e5d37f096d50d1f1ec7749e2b49d1c150b86f275a31815c284eb7a56e8f"} Jul 24 13:25:12.316215 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:12.316036 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" event={"ID":"05074f7e-d0b9-459d-8c68-326c2608af88","Type":"ContainerStarted","Data":"3a63201f2583c115170b15619540cb62ba808c452f2e3f70d88d2b44a9effd02"} Jul 24 13:25:12.337181 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:12.337126 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" podStartSLOduration=1.3371078490000001 podStartE2EDuration="1.337107849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:25:12.336637452 +0000 UTC m=+1928.339939557" watchObservedRunningTime="2026-07-24 13:25:12.337107849 +0000 UTC m=+1928.340409953" Jul 24 13:25:13.322478 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:13.322442 2554 generic.go:356] "Generic (PLEG): container finished" podID="05074f7e-d0b9-459d-8c68-326c2608af88" containerID="7bdc5e5d37f096d50d1f1ec7749e2b49d1c150b86f275a31815c284eb7a56e8f" exitCode=0 Jul 24 13:25:13.322974 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:13.322527 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" event={"ID":"05074f7e-d0b9-459d-8c68-326c2608af88","Type":"ContainerDied","Data":"7bdc5e5d37f096d50d1f1ec7749e2b49d1c150b86f275a31815c284eb7a56e8f"} Jul 24 13:25:13.692400 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:13.692308 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:14.457072 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:14.457046 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" Jul 24 13:25:14.532731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:14.532697 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05074f7e-d0b9-459d-8c68-326c2608af88-maas-api-service-ca\") pod \"05074f7e-d0b9-459d-8c68-326c2608af88\" (UID: \"05074f7e-d0b9-459d-8c68-326c2608af88\") " Jul 24 13:25:14.533063 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:14.533038 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05074f7e-d0b9-459d-8c68-326c2608af88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05074f7e-d0b9-459d-8c68-326c2608af88" (UID: "05074f7e-d0b9-459d-8c68-326c2608af88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:25:14.633183 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:14.633158 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05074f7e-d0b9-459d-8c68-326c2608af88-maas-api-service-ca\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:25:15.331788 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:15.331758 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" Jul 24 13:25:15.331951 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:15.331756 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc" event={"ID":"05074f7e-d0b9-459d-8c68-326c2608af88","Type":"ContainerDied","Data":"3a63201f2583c115170b15619540cb62ba808c452f2e3f70d88d2b44a9effd02"} Jul 24 13:25:15.331951 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:15.331878 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a63201f2583c115170b15619540cb62ba808c452f2e3f70d88d2b44a9effd02" Jul 24 13:25:21.873739 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:21.873704 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:21.906918 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:21.906887 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x"] Jul 24 13:25:21.907152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:21.907122 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" podUID="1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4" containerName="payload-processing" containerID="cri-o://d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c" gracePeriod=30 Jul 24 13:25:21.912451 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:21.912421 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj"] Jul 24 13:25:21.912850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:21.912803 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" podUID="7e244413-4359-49e4-8415-1e95789ee09f" containerName="payload-pre-processing" containerID="cri-o://782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c" gracePeriod=30 Jul 24 13:25:21.952444 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:21.952418 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:22.052623 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.052593 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:22.172115 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.172093 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:22.174775 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.174758 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:25:22.194138 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.194110 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hxb7\" (UniqueName: \"kubernetes.io/projected/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-kube-api-access-9hxb7\") pod \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\" (UID: \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\") " Jul 24 13:25:22.194256 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.194148 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf9ml\" (UniqueName: \"kubernetes.io/projected/7e244413-4359-49e4-8415-1e95789ee09f-kube-api-access-cf9ml\") pod \"7e244413-4359-49e4-8415-1e95789ee09f\" (UID: \"7e244413-4359-49e4-8415-1e95789ee09f\") " Jul 24 13:25:22.194256 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.194197 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-plugins-config-volume\") pod \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\" (UID: \"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4\") " Jul 24 13:25:22.194379 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.194286 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e244413-4359-49e4-8415-1e95789ee09f-plugins-config-volume\") pod \"7e244413-4359-49e4-8415-1e95789ee09f\" (UID: \"7e244413-4359-49e4-8415-1e95789ee09f\") " Jul 24 13:25:22.194651 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.194625 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4" (UID: "1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:25:22.194867 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.194837 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e244413-4359-49e4-8415-1e95789ee09f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e244413-4359-49e4-8415-1e95789ee09f" (UID: "7e244413-4359-49e4-8415-1e95789ee09f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:25:22.196503 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.196476 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e244413-4359-49e4-8415-1e95789ee09f-kube-api-access-cf9ml" (OuterVolumeSpecName: "kube-api-access-cf9ml") pod "7e244413-4359-49e4-8415-1e95789ee09f" (UID: "7e244413-4359-49e4-8415-1e95789ee09f"). InnerVolumeSpecName "kube-api-access-cf9ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:25:22.196623 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.196482 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-kube-api-access-9hxb7" (OuterVolumeSpecName: "kube-api-access-9hxb7") pod "1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4" (UID: "1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4"). InnerVolumeSpecName "kube-api-access-9hxb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:25:22.295207 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.295172 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e244413-4359-49e4-8415-1e95789ee09f-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:25:22.295207 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.295200 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hxb7\" (UniqueName: \"kubernetes.io/projected/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-kube-api-access-9hxb7\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:25:22.295207 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.295210 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9ml\" (UniqueName: \"kubernetes.io/projected/7e244413-4359-49e4-8415-1e95789ee09f-kube-api-access-cf9ml\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:25:22.295207 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.295218 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:25:22.361500 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.361459 2554 generic.go:356] "Generic (PLEG): container finished" podID="1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4" containerID="d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c" exitCode=0 Jul 24 13:25:22.361680 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.361502 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" event={"ID":"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4","Type":"ContainerDied","Data":"d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c"} Jul 24 13:25:22.361680 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.361554 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" event={"ID":"1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4","Type":"ContainerDied","Data":"59e1b1437032ea5c2660b612510152a4e3e72b51ccc2f851d295f289d322308e"} Jul 24 13:25:22.361680 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.361599 2554 scope.go:117] "RemoveContainer" containerID="d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c" Jul 24 13:25:22.361680 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.361512 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x" Jul 24 13:25:22.363142 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.363119 2554 generic.go:356] "Generic (PLEG): container finished" podID="7e244413-4359-49e4-8415-1e95789ee09f" containerID="782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c" exitCode=0 Jul 24 13:25:22.363270 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.363166 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" Jul 24 13:25:22.363270 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.363171 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" event={"ID":"7e244413-4359-49e4-8415-1e95789ee09f","Type":"ContainerDied","Data":"782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c"} Jul 24 13:25:22.363270 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.363204 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj" event={"ID":"7e244413-4359-49e4-8415-1e95789ee09f","Type":"ContainerDied","Data":"6655c86b54482304c7489d64e949bacda1e281a529e2d74a7a44cb6689ea178b"} Jul 24 13:25:22.370986 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.370964 2554 scope.go:117] "RemoveContainer" containerID="d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c" Jul 24 13:25:22.371322 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:25:22.371300 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c\": container with ID starting with d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c not found: ID does not exist" containerID="d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c" Jul 24 13:25:22.371422 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.371328 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c"} err="failed to get container status \"d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c\": rpc error: code = NotFound desc = could not find container \"d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c\": container with ID starting with d916abd417e4be0d3bdd39c964ee4b2cdc4fa87f86b1bbd7578631602575ae6c not found: ID does not exist" Jul 24 13:25:22.371422 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.371344 2554 scope.go:117] "RemoveContainer" containerID="782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c" Jul 24 13:25:22.379241 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.379223 2554 scope.go:117] "RemoveContainer" containerID="782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c" Jul 24 13:25:22.379509 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:25:22.379491 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c\": container with ID starting with 782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c not found: ID does not exist" containerID="782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c" Jul 24 13:25:22.379550 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.379516 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c"} err="failed to get container status \"782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c\": rpc error: code = NotFound desc = could not find container \"782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c\": container with ID starting with 782c5762af18c5b0a74d45959220cf93daf3367d97b315f62f3210669e20034c not found: ID does not exist" Jul 24 13:25:22.388411 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.388373 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x"] Jul 24 13:25:22.392401 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.392380 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1c83c54-65f658cbfd-27c7x"] Jul 24 13:25:22.405043 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.404987 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj"] Jul 24 13:25:22.411487 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.411468 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1c83c54-684795596d-wtrvj"] Jul 24 13:25:22.602749 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.602708 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4" path="/var/lib/kubelet/pods/1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4/volumes" Jul 24 13:25:22.603125 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:22.603105 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e244413-4359-49e4-8415-1e95789ee09f" path="/var/lib/kubelet/pods/7e244413-4359-49e4-8415-1e95789ee09f/volumes" Jul 24 13:25:24.952320 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:24.952291 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:25:26.438896 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:26.438860 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc"] Jul 24 13:25:26.439850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:26.439821 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:26.445229 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:26.445208 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1c83c54-5m5tc"] Jul 24 13:25:26.598371 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:26.598337 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05074f7e-d0b9-459d-8c68-326c2608af88" path="/var/lib/kubelet/pods/05074f7e-d0b9-459d-8c68-326c2608af88/volumes" Jul 24 13:25:26.834832 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:26.834805 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:33.314356 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:33.314308 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:33.415858 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:33.415819 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:33.605087 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:33.605005 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:33.837939 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:33.837905 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:34.003597 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:34.003495 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:34.110881 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:34.110849 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:34.203213 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:34.203172 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:40.550756 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:40.550721 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:40.651539 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:40.651510 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:40.752910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:40.752876 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:40.924907 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:40.924830 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:47.265728 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:47.265684 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:47.339040 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:47.339002 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:54.092352 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:54.092322 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:55.324615 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:55.324519 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:56.976682 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:56.976644 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:57.069079 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.069036 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:57.147023 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.146989 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:57.787584 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.787534 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:57.887023 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.886984 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:25:57.908307 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908269 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d"] Jul 24 13:25:57.908596 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908581 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4" containerName="payload-processing" Jul 24 13:25:57.908596 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908596 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4" containerName="payload-processing" Jul 24 13:25:57.908699 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908614 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05074f7e-d0b9-459d-8c68-326c2608af88" containerName="revoke-keys" Jul 24 13:25:57.908699 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908620 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05074f7e-d0b9-459d-8c68-326c2608af88" containerName="revoke-keys" Jul 24 13:25:57.908699 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908635 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e244413-4359-49e4-8415-1e95789ee09f" containerName="payload-pre-processing" Jul 24 13:25:57.908699 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908644 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e244413-4359-49e4-8415-1e95789ee09f" containerName="payload-pre-processing" Jul 24 13:25:57.908699 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908690 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff43e3d-55c9-4b8a-9d6d-9788e03d2be4" containerName="payload-processing" Jul 24 13:25:57.908872 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908700 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="05074f7e-d0b9-459d-8c68-326c2608af88" containerName="revoke-keys" Jul 24 13:25:57.908872 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.908711 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e244413-4359-49e4-8415-1e95789ee09f" containerName="payload-pre-processing" Jul 24 13:25:57.916347 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.916319 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:25:57.919017 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.918985 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-416b8ad5\"" Jul 24 13:25:57.920709 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.920688 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-416b8ad5-dockercfg-wmglt\"" Jul 24 13:25:57.923768 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.923178 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x"] Jul 24 13:25:57.926959 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.926940 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d"] Jul 24 13:25:57.927052 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.927044 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:25:57.935665 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:57.935637 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x"] Jul 24 13:25:58.015906 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.015877 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvbt\" (UniqueName: \"kubernetes.io/projected/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-kube-api-access-4lvbt\") pod \"payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d\" (UID: \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:25:58.016297 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.015929 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5grf\" (UniqueName: \"kubernetes.io/projected/887bd507-41f1-4889-984a-475612011dd1-kube-api-access-r5grf\") pod \"payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x\" (UID: \"887bd507-41f1-4889-984a-475612011dd1\") " pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:25:58.016297 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.016005 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d\" (UID: \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:25:58.016297 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.016053 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887bd507-41f1-4889-984a-475612011dd1-plugins-config-volume\") pod \"payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x\" (UID: \"887bd507-41f1-4889-984a-475612011dd1\") " pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:25:58.116751 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.116657 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvbt\" (UniqueName: \"kubernetes.io/projected/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-kube-api-access-4lvbt\") pod \"payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d\" (UID: \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:25:58.116751 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.116728 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5grf\" (UniqueName: \"kubernetes.io/projected/887bd507-41f1-4889-984a-475612011dd1-kube-api-access-r5grf\") pod \"payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x\" (UID: \"887bd507-41f1-4889-984a-475612011dd1\") " pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:25:58.116955 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.116774 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d\" (UID: \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:25:58.116955 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.116817 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887bd507-41f1-4889-984a-475612011dd1-plugins-config-volume\") pod \"payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x\" (UID: \"887bd507-41f1-4889-984a-475612011dd1\") " pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:25:58.117524 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.117505 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887bd507-41f1-4889-984a-475612011dd1-plugins-config-volume\") pod \"payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x\" (UID: \"887bd507-41f1-4889-984a-475612011dd1\") " pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:25:58.117599 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.117585 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d\" (UID: \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:25:58.127764 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.127738 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvbt\" (UniqueName: \"kubernetes.io/projected/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-kube-api-access-4lvbt\") pod \"payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d\" (UID: \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:25:58.127764 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.127748 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5grf\" (UniqueName: \"kubernetes.io/projected/887bd507-41f1-4889-984a-475612011dd1-kube-api-access-r5grf\") pod \"payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x\" (UID: \"887bd507-41f1-4889-984a-475612011dd1\") " pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:25:58.230414 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.230378 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:25:58.240492 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.240463 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:25:58.377056 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.377030 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d"] Jul 24 13:25:58.380206 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:25:58.380179 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259f3fd1_1929_46d9_ac9f_7dcbc5a69862.slice/crio-498bc3167f8759bcc2ee9113aef21208adb31e798ba1de6e3ed489c326274719 WatchSource:0}: Error finding container 498bc3167f8759bcc2ee9113aef21208adb31e798ba1de6e3ed489c326274719: Status 404 returned error can't find the container with id 498bc3167f8759bcc2ee9113aef21208adb31e798ba1de6e3ed489c326274719 Jul 24 13:25:58.396834 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.396810 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x"] Jul 24 13:25:58.399191 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:25:58.399169 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887bd507_41f1_4889_984a_475612011dd1.slice/crio-339e7aad3e2e531d797dcc5fec7a82b5f5025b93956e00209bfb19652fa0103a WatchSource:0}: Error finding container 339e7aad3e2e531d797dcc5fec7a82b5f5025b93956e00209bfb19652fa0103a: Status 404 returned error can't find the container with id 339e7aad3e2e531d797dcc5fec7a82b5f5025b93956e00209bfb19652fa0103a Jul 24 13:25:58.520456 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.520410 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" event={"ID":"259f3fd1-1929-46d9-ac9f-7dcbc5a69862","Type":"ContainerStarted","Data":"3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582"} Jul 24 13:25:58.520667 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.520484 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" event={"ID":"259f3fd1-1929-46d9-ac9f-7dcbc5a69862","Type":"ContainerStarted","Data":"498bc3167f8759bcc2ee9113aef21208adb31e798ba1de6e3ed489c326274719"} Jul 24 13:25:58.520667 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.520522 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:25:58.521927 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.521903 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" event={"ID":"887bd507-41f1-4889-984a-475612011dd1","Type":"ContainerStarted","Data":"650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb"} Jul 24 13:25:58.522028 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.521943 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" event={"ID":"887bd507-41f1-4889-984a-475612011dd1","Type":"ContainerStarted","Data":"339e7aad3e2e531d797dcc5fec7a82b5f5025b93956e00209bfb19652fa0103a"} Jul 24 13:25:58.522028 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.521974 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:25:58.538164 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.538109 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" podStartSLOduration=1.538092705 podStartE2EDuration="1.538092705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:25:58.537831173 +0000 UTC m=+1974.541133278" watchObservedRunningTime="2026-07-24 13:25:58.538092705 +0000 UTC m=+1974.541394812" Jul 24 13:25:58.555592 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:25:58.555513 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" podStartSLOduration=1.555492122 podStartE2EDuration="1.555492122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:25:58.554114456 +0000 UTC m=+1974.557416575" watchObservedRunningTime="2026-07-24 13:25:58.555492122 +0000 UTC m=+1974.558794226" Jul 24 13:26:01.563489 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:01.563456 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:26:03.554390 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:03.554350 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:26:05.503305 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:05.503272 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:26:09.530345 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:09.530313 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:26:09.530767 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:09.530390 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:26:09.713745 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:09.713705 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:26:10.486420 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:10.486387 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:26:10.587548 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:10.587517 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:26:17.650400 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:17.650364 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:26:27.453921 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:27.453883 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:26:39.556077 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:39.556042 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:26:41.244781 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.244743 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:26:41.261921 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.261895 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x"] Jul 24 13:26:41.262122 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.262100 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" podUID="887bd507-41f1-4889-984a-475612011dd1" containerName="payload-processing" containerID="cri-o://650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb" gracePeriod=30 Jul 24 13:26:41.271410 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.271383 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d"] Jul 24 13:26:41.271810 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.271778 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" podUID="259f3fd1-1929-46d9-ac9f-7dcbc5a69862" containerName="payload-pre-processing" containerID="cri-o://3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582" gracePeriod=30 Jul 24 13:26:41.341863 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.341812 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:26:41.509990 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.509968 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:26:41.536431 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.536409 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:26:41.582026 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.581995 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lvbt\" (UniqueName: \"kubernetes.io/projected/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-kube-api-access-4lvbt\") pod \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\" (UID: \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\") " Jul 24 13:26:41.582202 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.582079 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887bd507-41f1-4889-984a-475612011dd1-plugins-config-volume\") pod \"887bd507-41f1-4889-984a-475612011dd1\" (UID: \"887bd507-41f1-4889-984a-475612011dd1\") " Jul 24 13:26:41.582202 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.582127 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5grf\" (UniqueName: \"kubernetes.io/projected/887bd507-41f1-4889-984a-475612011dd1-kube-api-access-r5grf\") pod \"887bd507-41f1-4889-984a-475612011dd1\" (UID: \"887bd507-41f1-4889-984a-475612011dd1\") " Jul 24 13:26:41.582202 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.582143 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-plugins-config-volume\") pod \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\" (UID: \"259f3fd1-1929-46d9-ac9f-7dcbc5a69862\") " Jul 24 13:26:41.582522 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.582492 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887bd507-41f1-4889-984a-475612011dd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "887bd507-41f1-4889-984a-475612011dd1" (UID: "887bd507-41f1-4889-984a-475612011dd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:26:41.582644 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.582542 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "259f3fd1-1929-46d9-ac9f-7dcbc5a69862" (UID: "259f3fd1-1929-46d9-ac9f-7dcbc5a69862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:26:41.584380 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.584351 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-kube-api-access-4lvbt" (OuterVolumeSpecName: "kube-api-access-4lvbt") pod "259f3fd1-1929-46d9-ac9f-7dcbc5a69862" (UID: "259f3fd1-1929-46d9-ac9f-7dcbc5a69862"). InnerVolumeSpecName "kube-api-access-4lvbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:26:41.584464 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.584380 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887bd507-41f1-4889-984a-475612011dd1-kube-api-access-r5grf" (OuterVolumeSpecName: "kube-api-access-r5grf") pod "887bd507-41f1-4889-984a-475612011dd1" (UID: "887bd507-41f1-4889-984a-475612011dd1"). InnerVolumeSpecName "kube-api-access-r5grf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:26:41.683700 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.683657 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5grf\" (UniqueName: \"kubernetes.io/projected/887bd507-41f1-4889-984a-475612011dd1-kube-api-access-r5grf\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:26:41.683700 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.683691 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:26:41.683700 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.683700 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lvbt\" (UniqueName: \"kubernetes.io/projected/259f3fd1-1929-46d9-ac9f-7dcbc5a69862-kube-api-access-4lvbt\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:26:41.683700 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.683709 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887bd507-41f1-4889-984a-475612011dd1-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:26:41.711363 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.711328 2554 generic.go:356] "Generic (PLEG): container finished" podID="259f3fd1-1929-46d9-ac9f-7dcbc5a69862" containerID="3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582" exitCode=0 Jul 24 13:26:41.711526 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.711373 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" Jul 24 13:26:41.711526 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.711424 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" event={"ID":"259f3fd1-1929-46d9-ac9f-7dcbc5a69862","Type":"ContainerDied","Data":"3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582"} Jul 24 13:26:41.711526 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.711469 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d" event={"ID":"259f3fd1-1929-46d9-ac9f-7dcbc5a69862","Type":"ContainerDied","Data":"498bc3167f8759bcc2ee9113aef21208adb31e798ba1de6e3ed489c326274719"} Jul 24 13:26:41.711526 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.711492 2554 scope.go:117] "RemoveContainer" containerID="3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582" Jul 24 13:26:41.712782 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.712755 2554 generic.go:356] "Generic (PLEG): container finished" podID="887bd507-41f1-4889-984a-475612011dd1" containerID="650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb" exitCode=0 Jul 24 13:26:41.712891 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.712792 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" Jul 24 13:26:41.712891 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.712842 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" event={"ID":"887bd507-41f1-4889-984a-475612011dd1","Type":"ContainerDied","Data":"650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb"} Jul 24 13:26:41.712965 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.712895 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x" event={"ID":"887bd507-41f1-4889-984a-475612011dd1","Type":"ContainerDied","Data":"339e7aad3e2e531d797dcc5fec7a82b5f5025b93956e00209bfb19652fa0103a"} Jul 24 13:26:41.723398 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.723375 2554 scope.go:117] "RemoveContainer" containerID="3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582" Jul 24 13:26:41.723742 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:26:41.723720 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582\": container with ID starting with 3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582 not found: ID does not exist" containerID="3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582" Jul 24 13:26:41.723797 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.723752 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582"} err="failed to get container status \"3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582\": rpc error: code = NotFound desc = could not find container \"3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582\": container with ID starting with 3ce094745d8c0180f1198dfe6ccd2ad991b2f2f5a7b8588a74ca8b2c404c2582 not found: ID does not exist" Jul 24 13:26:41.723797 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.723774 2554 scope.go:117] "RemoveContainer" containerID="650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb" Jul 24 13:26:41.731545 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.731528 2554 scope.go:117] "RemoveContainer" containerID="650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb" Jul 24 13:26:41.731807 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:26:41.731791 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb\": container with ID starting with 650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb not found: ID does not exist" containerID="650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb" Jul 24 13:26:41.731846 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.731813 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb"} err="failed to get container status \"650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb\": rpc error: code = NotFound desc = could not find container \"650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb\": container with ID starting with 650384b1c504b267de52e20cff1e0614ab250c2b6e15afadba2b0d70df3540eb not found: ID does not exist" Jul 24 13:26:41.738796 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.738774 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x"] Jul 24 13:26:41.741339 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.741318 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-416b8ad5-6c7cdf9946-8259x"] Jul 24 13:26:41.756809 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.756746 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d"] Jul 24 13:26:41.763841 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:41.763820 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-416b8ad5-6584db6d56-drv9d"] Jul 24 13:26:42.598845 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:42.598816 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="259f3fd1-1929-46d9-ac9f-7dcbc5a69862" path="/var/lib/kubelet/pods/259f3fd1-1929-46d9-ac9f-7dcbc5a69862/volumes" Jul 24 13:26:42.599224 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:42.599129 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="887bd507-41f1-4889-984a-475612011dd1" path="/var/lib/kubelet/pods/887bd507-41f1-4889-984a-475612011dd1/volumes" Jul 24 13:26:45.719040 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:45.719011 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:26:49.026061 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:49.026029 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:26:49.949867 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:49.949838 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:26:54.060228 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:26:54.060192 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:27.899319 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:27.899238 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:30.179947 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:30.179911 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:32.479013 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:32.478980 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:38.901010 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:38.900981 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:39.004006 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:39.003972 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:39.206035 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:39.205956 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:39.436520 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:39.436490 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:40.549054 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:40.549018 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:40.663026 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:40.662998 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:40.762884 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:40.762850 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:41.522836 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:41.522796 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:41.623264 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:41.623227 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:41.722183 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:41.722152 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:43.102879 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:43.102841 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:43.159066 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:43.159031 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:27:50.114181 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:50.114147 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:50.232820 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:50.232787 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:50.823134 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:50.823101 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:53.058187 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:53.058150 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:27:57.980726 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:57.980694 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:58.095363 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:58.095332 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:58.212452 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:58.212419 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:27:58.766327 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:27:58.766288 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:04.619470 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:04.619431 2554 scope.go:117] "RemoveContainer" containerID="1f02256542ff12faf53669c19527dc06a95c9367711769df80e00706455feba4" Jul 24 13:28:08.737218 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:08.737182 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:09.102949 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:09.102918 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:30.107084 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:30.107045 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:38.325277 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:38.325244 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:38.564252 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:38.564220 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:38.764795 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:38.764717 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:39.127143 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:39.127111 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:39.358941 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:39.358908 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:41.983442 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:41.983409 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:42.118004 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:42.117965 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:42.238337 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:42.238238 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:42.563742 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:42.563706 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:42.946595 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:42.946477 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:43.063073 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.063040 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:43.072270 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.072240 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq"] Jul 24 13:28:43.072554 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.072542 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259f3fd1-1929-46d9-ac9f-7dcbc5a69862" containerName="payload-pre-processing" Jul 24 13:28:43.072624 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.072555 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="259f3fd1-1929-46d9-ac9f-7dcbc5a69862" containerName="payload-pre-processing" Jul 24 13:28:43.072624 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.072579 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887bd507-41f1-4889-984a-475612011dd1" containerName="payload-processing" Jul 24 13:28:43.072624 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.072585 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="887bd507-41f1-4889-984a-475612011dd1" containerName="payload-processing" Jul 24 13:28:43.072746 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.072645 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="259f3fd1-1929-46d9-ac9f-7dcbc5a69862" containerName="payload-pre-processing" Jul 24 13:28:43.072746 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.072652 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="887bd507-41f1-4889-984a-475612011dd1" containerName="payload-processing" Jul 24 13:28:43.075208 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.075194 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:28:43.078157 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.078138 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c72f4f\"" Jul 24 13:28:43.088751 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.088726 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq"] Jul 24 13:28:43.155365 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.155331 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5"] Jul 24 13:28:43.158329 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.158314 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:43.161551 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.161525 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c72f4f-dockercfg-pfv2t\"" Jul 24 13:28:43.172749 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.172731 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5"] Jul 24 13:28:43.226686 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.226593 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k68w2\" (UniqueName: \"kubernetes.io/projected/eb01ea89-2eea-4275-98f8-8823e5a94e24-kube-api-access-k68w2\") pod \"payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq\" (UID: \"eb01ea89-2eea-4275-98f8-8823e5a94e24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:28:43.226686 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.226665 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb01ea89-2eea-4275-98f8-8823e5a94e24-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq\" (UID: \"eb01ea89-2eea-4275-98f8-8823e5a94e24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:28:43.327429 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.327395 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb01ea89-2eea-4275-98f8-8823e5a94e24-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq\" (UID: \"eb01ea89-2eea-4275-98f8-8823e5a94e24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:28:43.327650 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.327487 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89cd925-0a9b-471e-b699-321a1f4670c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5\" (UID: \"a89cd925-0a9b-471e-b699-321a1f4670c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:43.327650 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.327533 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k68w2\" (UniqueName: \"kubernetes.io/projected/eb01ea89-2eea-4275-98f8-8823e5a94e24-kube-api-access-k68w2\") pod \"payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq\" (UID: \"eb01ea89-2eea-4275-98f8-8823e5a94e24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:28:43.327650 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.327577 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjj4\" (UniqueName: \"kubernetes.io/projected/a89cd925-0a9b-471e-b699-321a1f4670c9-kube-api-access-7fjj4\") pod \"payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5\" (UID: \"a89cd925-0a9b-471e-b699-321a1f4670c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:43.328133 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.328111 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb01ea89-2eea-4275-98f8-8823e5a94e24-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq\" (UID: \"eb01ea89-2eea-4275-98f8-8823e5a94e24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:28:43.341434 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.341412 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k68w2\" (UniqueName: \"kubernetes.io/projected/eb01ea89-2eea-4275-98f8-8823e5a94e24-kube-api-access-k68w2\") pod \"payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq\" (UID: \"eb01ea89-2eea-4275-98f8-8823e5a94e24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:28:43.385674 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.385639 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:28:43.428350 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.428306 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjj4\" (UniqueName: \"kubernetes.io/projected/a89cd925-0a9b-471e-b699-321a1f4670c9-kube-api-access-7fjj4\") pod \"payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5\" (UID: \"a89cd925-0a9b-471e-b699-321a1f4670c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:43.428548 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.428420 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89cd925-0a9b-471e-b699-321a1f4670c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5\" (UID: \"a89cd925-0a9b-471e-b699-321a1f4670c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:43.429313 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.429290 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89cd925-0a9b-471e-b699-321a1f4670c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5\" (UID: \"a89cd925-0a9b-471e-b699-321a1f4670c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:43.438597 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.438546 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjj4\" (UniqueName: \"kubernetes.io/projected/a89cd925-0a9b-471e-b699-321a1f4670c9-kube-api-access-7fjj4\") pod \"payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5\" (UID: \"a89cd925-0a9b-471e-b699-321a1f4670c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:43.468822 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.468795 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:43.527454 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.527199 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq"] Jul 24 13:28:43.620030 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:43.619998 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5"] Jul 24 13:28:43.622828 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:28:43.622802 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89cd925_0a9b_471e_b699_321a1f4670c9.slice/crio-f4cc22e078880b645eaabddfd693e83dc5c28f619c2cb50f41c7c4f0d6f0f6d9 WatchSource:0}: Error finding container f4cc22e078880b645eaabddfd693e83dc5c28f619c2cb50f41c7c4f0d6f0f6d9: Status 404 returned error can't find the container with id f4cc22e078880b645eaabddfd693e83dc5c28f619c2cb50f41c7c4f0d6f0f6d9 Jul 24 13:28:44.226867 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:44.226831 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" event={"ID":"a89cd925-0a9b-471e-b699-321a1f4670c9","Type":"ContainerStarted","Data":"6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150"} Jul 24 13:28:44.227359 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:44.226891 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" event={"ID":"a89cd925-0a9b-471e-b699-321a1f4670c9","Type":"ContainerStarted","Data":"f4cc22e078880b645eaabddfd693e83dc5c28f619c2cb50f41c7c4f0d6f0f6d9"} Jul 24 13:28:44.227359 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:44.226946 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:44.228471 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:44.228448 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" event={"ID":"eb01ea89-2eea-4275-98f8-8823e5a94e24","Type":"ContainerStarted","Data":"5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a"} Jul 24 13:28:44.228609 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:44.228487 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" event={"ID":"eb01ea89-2eea-4275-98f8-8823e5a94e24","Type":"ContainerStarted","Data":"7c8d19e0a1653dbad5199351320d7e23cd495b410f260d9fc41610f75aa6f9e7"} Jul 24 13:28:44.228609 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:44.228602 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:28:44.245698 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:44.245654 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" podStartSLOduration=1.245637749 podStartE2EDuration="1.245637749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:28:44.245272502 +0000 UTC m=+2140.248574606" watchObservedRunningTime="2026-07-24 13:28:44.245637749 +0000 UTC m=+2140.248939859" Jul 24 13:28:44.269689 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:44.269645 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" podStartSLOduration=1.26963165 podStartE2EDuration="1.26963165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:28:44.267743996 +0000 UTC m=+2140.271046105" watchObservedRunningTime="2026-07-24 13:28:44.26963165 +0000 UTC m=+2140.272933754" Jul 24 13:28:46.460301 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:46.460270 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:46.800872 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:46.800839 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:47.140134 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:47.140048 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:47.268602 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:47.268547 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:28:47.892007 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:47.891979 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:49.964180 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:49.964151 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:50.101327 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.101297 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:50.215633 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.215516 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:50.401186 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.401155 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:50.812469 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.812433 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:50.899368 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.899338 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c"] Jul 24 13:28:50.902804 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.902790 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:28:50.905399 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.905377 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b318c6\"" Jul 24 13:28:50.905937 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.905919 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f"] Jul 24 13:28:50.908945 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.908926 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:28:50.910283 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.910256 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:50.912322 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.912299 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b318c6-dockercfg-vkltc\"" Jul 24 13:28:50.913701 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.913680 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c"] Jul 24 13:28:50.919529 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.919504 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f"] Jul 24 13:28:50.994661 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.994621 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631a4ef4-a4eb-444e-8194-019cb1348baa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c\" (UID: \"631a4ef4-a4eb-444e-8194-019cb1348baa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:28:50.995076 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:50.994684 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2qs\" (UniqueName: \"kubernetes.io/projected/631a4ef4-a4eb-444e-8194-019cb1348baa-kube-api-access-jj2qs\") pod \"payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c\" (UID: \"631a4ef4-a4eb-444e-8194-019cb1348baa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:28:51.095351 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.095254 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2qs\" (UniqueName: \"kubernetes.io/projected/631a4ef4-a4eb-444e-8194-019cb1348baa-kube-api-access-jj2qs\") pod \"payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c\" (UID: \"631a4ef4-a4eb-444e-8194-019cb1348baa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:28:51.095351 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.095298 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c7335-1f01-4089-a2f7-32ee7e374200-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f\" (UID: \"6c1c7335-1f01-4089-a2f7-32ee7e374200\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:28:51.095351 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.095334 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgzff\" (UniqueName: \"kubernetes.io/projected/6c1c7335-1f01-4089-a2f7-32ee7e374200-kube-api-access-rgzff\") pod \"payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f\" (UID: \"6c1c7335-1f01-4089-a2f7-32ee7e374200\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:28:51.095623 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.095369 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631a4ef4-a4eb-444e-8194-019cb1348baa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c\" (UID: \"631a4ef4-a4eb-444e-8194-019cb1348baa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:28:51.096075 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.096057 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631a4ef4-a4eb-444e-8194-019cb1348baa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c\" (UID: \"631a4ef4-a4eb-444e-8194-019cb1348baa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:28:51.106127 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.106097 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2qs\" (UniqueName: \"kubernetes.io/projected/631a4ef4-a4eb-444e-8194-019cb1348baa-kube-api-access-jj2qs\") pod \"payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c\" (UID: \"631a4ef4-a4eb-444e-8194-019cb1348baa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:28:51.196061 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.196023 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c7335-1f01-4089-a2f7-32ee7e374200-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f\" (UID: \"6c1c7335-1f01-4089-a2f7-32ee7e374200\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:28:51.196252 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.196084 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzff\" (UniqueName: \"kubernetes.io/projected/6c1c7335-1f01-4089-a2f7-32ee7e374200-kube-api-access-rgzff\") pod \"payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f\" (UID: \"6c1c7335-1f01-4089-a2f7-32ee7e374200\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:28:51.196747 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.196728 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c7335-1f01-4089-a2f7-32ee7e374200-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f\" (UID: \"6c1c7335-1f01-4089-a2f7-32ee7e374200\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:28:51.208735 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.208715 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzff\" (UniqueName: \"kubernetes.io/projected/6c1c7335-1f01-4089-a2f7-32ee7e374200-kube-api-access-rgzff\") pod \"payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f\" (UID: \"6c1c7335-1f01-4089-a2f7-32ee7e374200\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:28:51.215553 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.215532 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:28:51.224448 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.224425 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:28:51.370045 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.370022 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c"] Jul 24 13:28:51.372498 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:28:51.372465 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631a4ef4_a4eb_444e_8194_019cb1348baa.slice/crio-d96da145621f65ff8f0f301dbce98b74ef33e8437cf04034f3523d7fa3a014c8 WatchSource:0}: Error finding container d96da145621f65ff8f0f301dbce98b74ef33e8437cf04034f3523d7fa3a014c8: Status 404 returned error can't find the container with id d96da145621f65ff8f0f301dbce98b74ef33e8437cf04034f3523d7fa3a014c8 Jul 24 13:28:51.402899 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:51.402810 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f"] Jul 24 13:28:51.405263 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:28:51.405228 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1c7335_1f01_4089_a2f7_32ee7e374200.slice/crio-53e5601f7f1e09163105626b9775e3f7aef53a661de0f640f7711101856e0ef5 WatchSource:0}: Error finding container 53e5601f7f1e09163105626b9775e3f7aef53a661de0f640f7711101856e0ef5: Status 404 returned error can't find the container with id 53e5601f7f1e09163105626b9775e3f7aef53a661de0f640f7711101856e0ef5 Jul 24 13:28:52.267215 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:52.267178 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" event={"ID":"6c1c7335-1f01-4089-a2f7-32ee7e374200","Type":"ContainerStarted","Data":"d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b"} Jul 24 13:28:52.267654 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:52.267227 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" event={"ID":"6c1c7335-1f01-4089-a2f7-32ee7e374200","Type":"ContainerStarted","Data":"53e5601f7f1e09163105626b9775e3f7aef53a661de0f640f7711101856e0ef5"} Jul 24 13:28:52.267654 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:52.267281 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:28:52.268704 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:52.268681 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" event={"ID":"631a4ef4-a4eb-444e-8194-019cb1348baa","Type":"ContainerStarted","Data":"18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896"} Jul 24 13:28:52.268799 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:52.268719 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" event={"ID":"631a4ef4-a4eb-444e-8194-019cb1348baa","Type":"ContainerStarted","Data":"d96da145621f65ff8f0f301dbce98b74ef33e8437cf04034f3523d7fa3a014c8"} Jul 24 13:28:52.268799 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:52.268755 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:28:52.287439 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:52.287400 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" podStartSLOduration=2.287388697 podStartE2EDuration="2.287388697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:28:52.285875788 +0000 UTC m=+2148.289177892" watchObservedRunningTime="2026-07-24 13:28:52.287388697 +0000 UTC m=+2148.290690800" Jul 24 13:28:52.304160 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:52.304120 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" podStartSLOduration=2.304107519 podStartE2EDuration="2.304107519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:28:52.302825224 +0000 UTC m=+2148.306127352" watchObservedRunningTime="2026-07-24 13:28:52.304107519 +0000 UTC m=+2148.307409624" Jul 24 13:28:54.478637 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:54.478587 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:28:55.235962 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:55.235932 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:28:55.236142 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:28:55.236002 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:29:02.771117 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:02.771077 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:29:03.275324 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:03.275292 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:29:03.275504 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:03.275357 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:29:04.637214 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:04.637183 2554 scope.go:117] "RemoveContainer" containerID="160aad7dd893bfd3be84c3c223109c000ec23544795a5325889112ff6a9dabfb" Jul 24 13:29:09.332330 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.332299 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx"] Jul 24 13:29:09.335659 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.335642 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.340965 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.340942 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c72f4f\"/\"default-dockercfg-wxtwx\"" Jul 24 13:29:09.341119 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.341013 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c72f4f\"/\"test-model-c72f4f-kserve-self-signed-certs\"" Jul 24 13:29:09.341269 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.341166 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c72f4f\"/\"openshift-service-ca.crt\"" Jul 24 13:29:09.342849 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.342830 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c72f4f\"/\"kube-root-ca.crt\"" Jul 24 13:29:09.366731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.366699 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx"] Jul 24 13:29:09.442139 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.442100 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-model-cache\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.442386 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.442170 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-dshm\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.442386 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.442212 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tls-certs\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.442386 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.442236 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tmp-dir\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.442386 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.442257 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-home\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.442386 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.442297 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ch8r\" (UniqueName: \"kubernetes.io/projected/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-kube-api-access-4ch8r\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.446077 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.446052 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:29:09.543283 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.543244 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ch8r\" (UniqueName: \"kubernetes.io/projected/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-kube-api-access-4ch8r\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.543630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.543297 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-model-cache\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.543630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.543336 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-dshm\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.543630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.543354 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tls-certs\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.543630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.543374 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tmp-dir\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.543630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.543393 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-home\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.543922 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.543900 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-home\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.543991 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.543922 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-model-cache\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.543991 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.543977 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tmp-dir\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.545929 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.545906 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-dshm\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.546184 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.546168 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tls-certs\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.553847 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.553823 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ch8r\" (UniqueName: \"kubernetes.io/projected/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-kube-api-access-4ch8r\") pod \"test-model-c72f4f-kserve-96c55986f-cfpdx\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.645205 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.645114 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:09.790237 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:09.790211 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx"] Jul 24 13:29:09.792133 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:29:09.792109 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb0d7cf3_8da5_4d19_863b_9d3a98f47181.slice/crio-cc24286f45b92bec4cdd12cd4c76f7c36ebb4770814dbe4d00008f6f9bf35e4d WatchSource:0}: Error finding container cc24286f45b92bec4cdd12cd4c76f7c36ebb4770814dbe4d00008f6f9bf35e4d: Status 404 returned error can't find the container with id cc24286f45b92bec4cdd12cd4c76f7c36ebb4770814dbe4d00008f6f9bf35e4d Jul 24 13:29:10.352686 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:10.352646 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" event={"ID":"fb0d7cf3-8da5-4d19-863b-9d3a98f47181","Type":"ContainerStarted","Data":"fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430"} Jul 24 13:29:10.352686 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:10.352695 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" event={"ID":"fb0d7cf3-8da5-4d19-863b-9d3a98f47181","Type":"ContainerStarted","Data":"cc24286f45b92bec4cdd12cd4c76f7c36ebb4770814dbe4d00008f6f9bf35e4d"} Jul 24 13:29:10.373291 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:10.373242 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" podStartSLOduration=1.373226087 podStartE2EDuration="1.373226087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:29:10.372534729 +0000 UTC m=+2166.375836834" watchObservedRunningTime="2026-07-24 13:29:10.373226087 +0000 UTC m=+2166.376528190" Jul 24 13:29:12.969052 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:12.969013 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:29:19.645681 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:19.645638 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:19.646138 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:19.645914 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:19.658516 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:19.658495 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:20.408130 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:20.408104 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:29:32.152172 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:32.152140 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:29:36.362024 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:36.361986 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:29:49.074032 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:29:49.073993 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:30:11.036768 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:11.036712 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:11.109330 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:11.109297 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:14.396734 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:14.396703 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx"] Jul 24 13:30:14.421445 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:14.421412 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:15.396673 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:15.396634 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:21.558302 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:21.558266 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:30:26.266696 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.266627 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:26.290374 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.290338 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq"] Jul 24 13:30:26.290618 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.290591 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" podUID="eb01ea89-2eea-4275-98f8-8823e5a94e24" containerName="payload-processing" containerID="cri-o://5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a" gracePeriod=30 Jul 24 13:30:26.300756 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.300729 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5"] Jul 24 13:30:26.300988 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.300965 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" podUID="a89cd925-0a9b-471e-b699-321a1f4670c9" containerName="payload-pre-processing" containerID="cri-o://6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150" gracePeriod=30 Jul 24 13:30:26.362172 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.362125 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:26.573887 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.573860 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:30:26.578453 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.578428 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:30:26.685494 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.685459 2554 generic.go:356] "Generic (PLEG): container finished" podID="a89cd925-0a9b-471e-b699-321a1f4670c9" containerID="6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150" exitCode=0 Jul 24 13:30:26.685711 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.685512 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" Jul 24 13:30:26.685711 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.685532 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" event={"ID":"a89cd925-0a9b-471e-b699-321a1f4670c9","Type":"ContainerDied","Data":"6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150"} Jul 24 13:30:26.685711 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.685609 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5" event={"ID":"a89cd925-0a9b-471e-b699-321a1f4670c9","Type":"ContainerDied","Data":"f4cc22e078880b645eaabddfd693e83dc5c28f619c2cb50f41c7c4f0d6f0f6d9"} Jul 24 13:30:26.685711 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.685637 2554 scope.go:117] "RemoveContainer" containerID="6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150" Jul 24 13:30:26.686796 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.686769 2554 generic.go:356] "Generic (PLEG): container finished" podID="eb01ea89-2eea-4275-98f8-8823e5a94e24" containerID="5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a" exitCode=0 Jul 24 13:30:26.686915 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.686800 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" Jul 24 13:30:26.686915 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.686804 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" event={"ID":"eb01ea89-2eea-4275-98f8-8823e5a94e24","Type":"ContainerDied","Data":"5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a"} Jul 24 13:30:26.686915 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.686847 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq" event={"ID":"eb01ea89-2eea-4275-98f8-8823e5a94e24","Type":"ContainerDied","Data":"7c8d19e0a1653dbad5199351320d7e23cd495b410f260d9fc41610f75aa6f9e7"} Jul 24 13:30:26.694087 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.694071 2554 scope.go:117] "RemoveContainer" containerID="6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150" Jul 24 13:30:26.694393 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:30:26.694375 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150\": container with ID starting with 6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150 not found: ID does not exist" containerID="6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150" Jul 24 13:30:26.694450 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.694401 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150"} err="failed to get container status \"6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150\": rpc error: code = NotFound desc = could not find container \"6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150\": container with ID starting with 6445b39da065a60f9deb8e133d224036a84ac72838f2de5a58772ce43e9bc150 not found: ID does not exist" Jul 24 13:30:26.694450 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.694418 2554 scope.go:117] "RemoveContainer" containerID="5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a" Jul 24 13:30:26.702058 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.702040 2554 scope.go:117] "RemoveContainer" containerID="5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a" Jul 24 13:30:26.702314 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:30:26.702294 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a\": container with ID starting with 5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a not found: ID does not exist" containerID="5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a" Jul 24 13:30:26.702377 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.702324 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a"} err="failed to get container status \"5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a\": rpc error: code = NotFound desc = could not find container \"5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a\": container with ID starting with 5ae10f5666139f78c599dd8c757b1a9c7d756108162d3f079dab68ce3dd03c4a not found: ID does not exist" Jul 24 13:30:26.727677 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.727657 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fjj4\" (UniqueName: \"kubernetes.io/projected/a89cd925-0a9b-471e-b699-321a1f4670c9-kube-api-access-7fjj4\") pod \"a89cd925-0a9b-471e-b699-321a1f4670c9\" (UID: \"a89cd925-0a9b-471e-b699-321a1f4670c9\") " Jul 24 13:30:26.727800 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.727704 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb01ea89-2eea-4275-98f8-8823e5a94e24-plugins-config-volume\") pod \"eb01ea89-2eea-4275-98f8-8823e5a94e24\" (UID: \"eb01ea89-2eea-4275-98f8-8823e5a94e24\") " Jul 24 13:30:26.727800 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.727754 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89cd925-0a9b-471e-b699-321a1f4670c9-plugins-config-volume\") pod \"a89cd925-0a9b-471e-b699-321a1f4670c9\" (UID: \"a89cd925-0a9b-471e-b699-321a1f4670c9\") " Jul 24 13:30:26.727800 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.727793 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k68w2\" (UniqueName: \"kubernetes.io/projected/eb01ea89-2eea-4275-98f8-8823e5a94e24-kube-api-access-k68w2\") pod \"eb01ea89-2eea-4275-98f8-8823e5a94e24\" (UID: \"eb01ea89-2eea-4275-98f8-8823e5a94e24\") " Jul 24 13:30:26.728101 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.728076 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb01ea89-2eea-4275-98f8-8823e5a94e24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb01ea89-2eea-4275-98f8-8823e5a94e24" (UID: "eb01ea89-2eea-4275-98f8-8823e5a94e24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:30:26.728184 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.728075 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89cd925-0a9b-471e-b699-321a1f4670c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a89cd925-0a9b-471e-b699-321a1f4670c9" (UID: "a89cd925-0a9b-471e-b699-321a1f4670c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:30:26.729936 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.729911 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb01ea89-2eea-4275-98f8-8823e5a94e24-kube-api-access-k68w2" (OuterVolumeSpecName: "kube-api-access-k68w2") pod "eb01ea89-2eea-4275-98f8-8823e5a94e24" (UID: "eb01ea89-2eea-4275-98f8-8823e5a94e24"). InnerVolumeSpecName "kube-api-access-k68w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:30:26.730035 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.729935 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89cd925-0a9b-471e-b699-321a1f4670c9-kube-api-access-7fjj4" (OuterVolumeSpecName: "kube-api-access-7fjj4") pod "a89cd925-0a9b-471e-b699-321a1f4670c9" (UID: "a89cd925-0a9b-471e-b699-321a1f4670c9"). InnerVolumeSpecName "kube-api-access-7fjj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:30:26.828731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.828649 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb01ea89-2eea-4275-98f8-8823e5a94e24-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:26.828731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.828676 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89cd925-0a9b-471e-b699-321a1f4670c9-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:26.828731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.828686 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k68w2\" (UniqueName: \"kubernetes.io/projected/eb01ea89-2eea-4275-98f8-8823e5a94e24-kube-api-access-k68w2\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:26.828731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:26.828694 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fjj4\" (UniqueName: \"kubernetes.io/projected/a89cd925-0a9b-471e-b699-321a1f4670c9-kube-api-access-7fjj4\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:27.014382 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:27.014345 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5"] Jul 24 13:30:27.019756 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:27.019731 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c72f4f-75894c67b-kcbk5"] Jul 24 13:30:27.032515 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:27.032488 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq"] Jul 24 13:30:27.037136 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:27.037116 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c72f4f-675d4c845d-g5vdq"] Jul 24 13:30:28.598233 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:28.598189 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a89cd925-0a9b-471e-b699-321a1f4670c9" path="/var/lib/kubelet/pods/a89cd925-0a9b-471e-b699-321a1f4670c9/volumes" Jul 24 13:30:28.598747 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:28.598730 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb01ea89-2eea-4275-98f8-8823e5a94e24" path="/var/lib/kubelet/pods/eb01ea89-2eea-4275-98f8-8823e5a94e24/volumes" Jul 24 13:30:29.416361 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.416317 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" podUID="fb0d7cf3-8da5-4d19-863b-9d3a98f47181" containerName="main" containerID="cri-o://fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430" gracePeriod=45 Jul 24 13:30:29.664687 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.664662 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:30:29.703237 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.703151 2554 generic.go:356] "Generic (PLEG): container finished" podID="fb0d7cf3-8da5-4d19-863b-9d3a98f47181" containerID="fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430" exitCode=0 Jul 24 13:30:29.703237 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.703190 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" Jul 24 13:30:29.703237 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.703225 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" event={"ID":"fb0d7cf3-8da5-4d19-863b-9d3a98f47181","Type":"ContainerDied","Data":"fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430"} Jul 24 13:30:29.703534 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.703259 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx" event={"ID":"fb0d7cf3-8da5-4d19-863b-9d3a98f47181","Type":"ContainerDied","Data":"cc24286f45b92bec4cdd12cd4c76f7c36ebb4770814dbe4d00008f6f9bf35e4d"} Jul 24 13:30:29.703534 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.703278 2554 scope.go:117] "RemoveContainer" containerID="fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430" Jul 24 13:30:29.711845 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.711830 2554 scope.go:117] "RemoveContainer" containerID="fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430" Jul 24 13:30:29.712129 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:30:29.712109 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430\": container with ID starting with fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430 not found: ID does not exist" containerID="fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430" Jul 24 13:30:29.712217 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.712138 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430"} err="failed to get container status \"fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430\": rpc error: code = NotFound desc = could not find container \"fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430\": container with ID starting with fc31283c96a22c6ec0461ec06a97bf5cbad3c904d4d70c50fa5446398cce8430 not found: ID does not exist" Jul 24 13:30:29.752227 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.752195 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ch8r\" (UniqueName: \"kubernetes.io/projected/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-kube-api-access-4ch8r\") pod \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " Jul 24 13:30:29.752389 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.752238 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-home\") pod \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " Jul 24 13:30:29.752389 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.752277 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tmp-dir\") pod \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " Jul 24 13:30:29.752482 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.752441 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tls-certs\") pod \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " Jul 24 13:30:29.752537 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.752512 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb0d7cf3-8da5-4d19-863b-9d3a98f47181" (UID: "fb0d7cf3-8da5-4d19-863b-9d3a98f47181"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:30:29.752537 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.752526 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-home" (OuterVolumeSpecName: "home") pod "fb0d7cf3-8da5-4d19-863b-9d3a98f47181" (UID: "fb0d7cf3-8da5-4d19-863b-9d3a98f47181"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:30:29.752671 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.752530 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-dshm\") pod \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " Jul 24 13:30:29.752671 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.752630 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-model-cache\") pod \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\" (UID: \"fb0d7cf3-8da5-4d19-863b-9d3a98f47181\") " Jul 24 13:30:29.752910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.752885 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb0d7cf3-8da5-4d19-863b-9d3a98f47181" (UID: "fb0d7cf3-8da5-4d19-863b-9d3a98f47181"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:30:29.753027 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.753010 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-model-cache\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:29.753262 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.753035 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-home\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:29.753262 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.753160 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tmp-dir\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:29.754671 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.754638 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-kube-api-access-4ch8r" (OuterVolumeSpecName: "kube-api-access-4ch8r") pod "fb0d7cf3-8da5-4d19-863b-9d3a98f47181" (UID: "fb0d7cf3-8da5-4d19-863b-9d3a98f47181"). InnerVolumeSpecName "kube-api-access-4ch8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:30:29.754779 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.754678 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-dshm" (OuterVolumeSpecName: "dshm") pod "fb0d7cf3-8da5-4d19-863b-9d3a98f47181" (UID: "fb0d7cf3-8da5-4d19-863b-9d3a98f47181"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:30:29.754779 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.754705 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb0d7cf3-8da5-4d19-863b-9d3a98f47181" (UID: "fb0d7cf3-8da5-4d19-863b-9d3a98f47181"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:30:29.854434 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.854398 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-tls-certs\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:29.854434 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.854427 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-dshm\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:29.854434 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:29.854435 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ch8r\" (UniqueName: \"kubernetes.io/projected/fb0d7cf3-8da5-4d19-863b-9d3a98f47181-kube-api-access-4ch8r\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:30.033576 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:30.031947 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx"] Jul 24 13:30:30.036251 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:30.036224 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c72f4f/test-model-c72f4f-kserve-96c55986f-cfpdx"] Jul 24 13:30:30.597868 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:30.597837 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb0d7cf3-8da5-4d19-863b-9d3a98f47181" path="/var/lib/kubelet/pods/fb0d7cf3-8da5-4d19-863b-9d3a98f47181/volumes" Jul 24 13:30:30.788965 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:30.788928 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:31.049703 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:31.049675 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:31.156254 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:31.156222 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:30:37.973102 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:37.973059 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:38.075685 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:38.075655 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:38.299395 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:38.299368 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:38.539731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:38.539699 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:49.213728 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.213689 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:49.249251 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.249221 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f"] Jul 24 13:30:49.249448 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.249425 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" podUID="6c1c7335-1f01-4089-a2f7-32ee7e374200" containerName="payload-processing" containerID="cri-o://d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b" gracePeriod=30 Jul 24 13:30:49.258399 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.257839 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c"] Jul 24 13:30:49.258399 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.258057 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" podUID="631a4ef4-a4eb-444e-8194-019cb1348baa" containerName="payload-pre-processing" containerID="cri-o://18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896" gracePeriod=30 Jul 24 13:30:49.299449 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.299422 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:49.464267 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.464186 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:49.524302 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.524241 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:30:49.527947 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.527927 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:30:49.619668 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.619635 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c7335-1f01-4089-a2f7-32ee7e374200-plugins-config-volume\") pod \"6c1c7335-1f01-4089-a2f7-32ee7e374200\" (UID: \"6c1c7335-1f01-4089-a2f7-32ee7e374200\") " Jul 24 13:30:49.619850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.619690 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631a4ef4-a4eb-444e-8194-019cb1348baa-plugins-config-volume\") pod \"631a4ef4-a4eb-444e-8194-019cb1348baa\" (UID: \"631a4ef4-a4eb-444e-8194-019cb1348baa\") " Jul 24 13:30:49.619850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.619715 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj2qs\" (UniqueName: \"kubernetes.io/projected/631a4ef4-a4eb-444e-8194-019cb1348baa-kube-api-access-jj2qs\") pod \"631a4ef4-a4eb-444e-8194-019cb1348baa\" (UID: \"631a4ef4-a4eb-444e-8194-019cb1348baa\") " Jul 24 13:30:49.619850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.619749 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgzff\" (UniqueName: \"kubernetes.io/projected/6c1c7335-1f01-4089-a2f7-32ee7e374200-kube-api-access-rgzff\") pod \"6c1c7335-1f01-4089-a2f7-32ee7e374200\" (UID: \"6c1c7335-1f01-4089-a2f7-32ee7e374200\") " Jul 24 13:30:49.620037 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.620007 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1c7335-1f01-4089-a2f7-32ee7e374200-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c1c7335-1f01-4089-a2f7-32ee7e374200" (UID: "6c1c7335-1f01-4089-a2f7-32ee7e374200"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:30:49.620118 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.620098 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631a4ef4-a4eb-444e-8194-019cb1348baa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "631a4ef4-a4eb-444e-8194-019cb1348baa" (UID: "631a4ef4-a4eb-444e-8194-019cb1348baa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:30:49.621965 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.621943 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1c7335-1f01-4089-a2f7-32ee7e374200-kube-api-access-rgzff" (OuterVolumeSpecName: "kube-api-access-rgzff") pod "6c1c7335-1f01-4089-a2f7-32ee7e374200" (UID: "6c1c7335-1f01-4089-a2f7-32ee7e374200"). InnerVolumeSpecName "kube-api-access-rgzff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:30:49.622032 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.622019 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631a4ef4-a4eb-444e-8194-019cb1348baa-kube-api-access-jj2qs" (OuterVolumeSpecName: "kube-api-access-jj2qs") pod "631a4ef4-a4eb-444e-8194-019cb1348baa" (UID: "631a4ef4-a4eb-444e-8194-019cb1348baa"). InnerVolumeSpecName "kube-api-access-jj2qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:30:49.721195 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.721110 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1c7335-1f01-4089-a2f7-32ee7e374200-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:49.721195 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.721139 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631a4ef4-a4eb-444e-8194-019cb1348baa-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:49.721195 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.721152 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2qs\" (UniqueName: \"kubernetes.io/projected/631a4ef4-a4eb-444e-8194-019cb1348baa-kube-api-access-jj2qs\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:49.721195 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.721161 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgzff\" (UniqueName: \"kubernetes.io/projected/6c1c7335-1f01-4089-a2f7-32ee7e374200-kube-api-access-rgzff\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:30:49.795860 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.795827 2554 generic.go:356] "Generic (PLEG): container finished" podID="6c1c7335-1f01-4089-a2f7-32ee7e374200" containerID="d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b" exitCode=0 Jul 24 13:30:49.796044 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.795885 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" Jul 24 13:30:49.796044 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.795896 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" event={"ID":"6c1c7335-1f01-4089-a2f7-32ee7e374200","Type":"ContainerDied","Data":"d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b"} Jul 24 13:30:49.796044 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.795965 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f" event={"ID":"6c1c7335-1f01-4089-a2f7-32ee7e374200","Type":"ContainerDied","Data":"53e5601f7f1e09163105626b9775e3f7aef53a661de0f640f7711101856e0ef5"} Jul 24 13:30:49.796044 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.795987 2554 scope.go:117] "RemoveContainer" containerID="d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b" Jul 24 13:30:49.797234 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.797209 2554 generic.go:356] "Generic (PLEG): container finished" podID="631a4ef4-a4eb-444e-8194-019cb1348baa" containerID="18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896" exitCode=0 Jul 24 13:30:49.797364 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.797274 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" Jul 24 13:30:49.797364 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.797289 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" event={"ID":"631a4ef4-a4eb-444e-8194-019cb1348baa","Type":"ContainerDied","Data":"18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896"} Jul 24 13:30:49.797364 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.797330 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c" event={"ID":"631a4ef4-a4eb-444e-8194-019cb1348baa","Type":"ContainerDied","Data":"d96da145621f65ff8f0f301dbce98b74ef33e8437cf04034f3523d7fa3a014c8"} Jul 24 13:30:49.804795 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.804774 2554 scope.go:117] "RemoveContainer" containerID="d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b" Jul 24 13:30:49.805055 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:30:49.805035 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b\": container with ID starting with d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b not found: ID does not exist" containerID="d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b" Jul 24 13:30:49.805142 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.805064 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b"} err="failed to get container status \"d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b\": rpc error: code = NotFound desc = could not find container \"d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b\": container with ID starting with d296184713fe280375f174b9b49259e0ebbedb31f5e942284255a9bd0488ff8b not found: ID does not exist" Jul 24 13:30:49.805142 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.805086 2554 scope.go:117] "RemoveContainer" containerID="18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896" Jul 24 13:30:49.813234 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.813216 2554 scope.go:117] "RemoveContainer" containerID="18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896" Jul 24 13:30:49.813465 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:30:49.813446 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896\": container with ID starting with 18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896 not found: ID does not exist" containerID="18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896" Jul 24 13:30:49.813532 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.813476 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896"} err="failed to get container status \"18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896\": rpc error: code = NotFound desc = could not find container \"18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896\": container with ID starting with 18559fd2c8dac3c3878889d0276a335f2e62967c44faf674ad5f7e41561a1896 not found: ID does not exist" Jul 24 13:30:49.828750 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.828730 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c"] Jul 24 13:30:49.831839 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.831817 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b318c6-68bfd5448-8t96c"] Jul 24 13:30:49.844973 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.844953 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f"] Jul 24 13:30:49.849370 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:49.849351 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b318c6-74d87446c8-4fw7f"] Jul 24 13:30:50.598571 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:50.598527 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="631a4ef4-a4eb-444e-8194-019cb1348baa" path="/var/lib/kubelet/pods/631a4ef4-a4eb-444e-8194-019cb1348baa/volumes" Jul 24 13:30:50.598953 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:50.598900 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1c7335-1f01-4089-a2f7-32ee7e374200" path="/var/lib/kubelet/pods/6c1c7335-1f01-4089-a2f7-32ee7e374200/volumes" Jul 24 13:30:52.051699 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:52.051668 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:30:53.791140 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:53.791105 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:30:54.220932 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:30:54.220848 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:00.576452 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:00.576421 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:03.619363 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:03.619322 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:04.768211 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:04.768176 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:04.862061 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:04.862019 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:05.051929 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.051903 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:05.150941 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.150913 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:05.759004 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.758974 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:05.856820 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.856788 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:05.875807 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.875779 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn"] Jul 24 13:31:05.876062 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876051 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb01ea89-2eea-4275-98f8-8823e5a94e24" containerName="payload-processing" Jul 24 13:31:05.876107 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876064 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb01ea89-2eea-4275-98f8-8823e5a94e24" containerName="payload-processing" Jul 24 13:31:05.876107 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876073 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89cd925-0a9b-471e-b699-321a1f4670c9" containerName="payload-pre-processing" Jul 24 13:31:05.876107 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876078 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89cd925-0a9b-471e-b699-321a1f4670c9" containerName="payload-pre-processing" Jul 24 13:31:05.876107 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876088 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631a4ef4-a4eb-444e-8194-019cb1348baa" containerName="payload-pre-processing" Jul 24 13:31:05.876107 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876093 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="631a4ef4-a4eb-444e-8194-019cb1348baa" containerName="payload-pre-processing" Jul 24 13:31:05.876265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876110 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1c7335-1f01-4089-a2f7-32ee7e374200" containerName="payload-processing" Jul 24 13:31:05.876265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876115 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1c7335-1f01-4089-a2f7-32ee7e374200" containerName="payload-processing" Jul 24 13:31:05.876265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876121 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0d7cf3-8da5-4d19-863b-9d3a98f47181" containerName="main" Jul 24 13:31:05.876265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876126 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0d7cf3-8da5-4d19-863b-9d3a98f47181" containerName="main" Jul 24 13:31:05.876265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876170 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb01ea89-2eea-4275-98f8-8823e5a94e24" containerName="payload-processing" Jul 24 13:31:05.876265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876178 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0d7cf3-8da5-4d19-863b-9d3a98f47181" containerName="main" Jul 24 13:31:05.876265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876183 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1c7335-1f01-4089-a2f7-32ee7e374200" containerName="payload-processing" Jul 24 13:31:05.876265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876189 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a89cd925-0a9b-471e-b699-321a1f4670c9" containerName="payload-pre-processing" Jul 24 13:31:05.876265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.876194 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="631a4ef4-a4eb-444e-8194-019cb1348baa" containerName="payload-pre-processing" Jul 24 13:31:05.880263 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.880247 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:05.882908 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.882884 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e530f2\"" Jul 24 13:31:05.890869 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.890845 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn"] Jul 24 13:31:05.897940 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.897916 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v"] Jul 24 13:31:05.901209 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.901191 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:05.904401 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.904382 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e530f2-dockercfg-8xqxm\"" Jul 24 13:31:05.911495 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.911473 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v"] Jul 24 13:31:05.967950 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.967914 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhjj\" (UniqueName: \"kubernetes.io/projected/9de1d577-2e4c-4d73-bc42-eca44becba5e-kube-api-access-jrhjj\") pod \"payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn\" (UID: \"9de1d577-2e4c-4d73-bc42-eca44becba5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:05.968134 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.967969 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52dd9\" (UniqueName: \"kubernetes.io/projected/a8a66df6-d758-46a6-af02-909873244fca-kube-api-access-52dd9\") pod \"payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v\" (UID: \"a8a66df6-d758-46a6-af02-909873244fca\") " pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:05.968134 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.968029 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a66df6-d758-46a6-af02-909873244fca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v\" (UID: \"a8a66df6-d758-46a6-af02-909873244fca\") " pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:05.968134 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:05.968052 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de1d577-2e4c-4d73-bc42-eca44becba5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn\" (UID: \"9de1d577-2e4c-4d73-bc42-eca44becba5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:06.069207 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.069176 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52dd9\" (UniqueName: \"kubernetes.io/projected/a8a66df6-d758-46a6-af02-909873244fca-kube-api-access-52dd9\") pod \"payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v\" (UID: \"a8a66df6-d758-46a6-af02-909873244fca\") " pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:06.069375 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.069223 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a66df6-d758-46a6-af02-909873244fca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v\" (UID: \"a8a66df6-d758-46a6-af02-909873244fca\") " pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:06.069375 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.069246 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de1d577-2e4c-4d73-bc42-eca44becba5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn\" (UID: \"9de1d577-2e4c-4d73-bc42-eca44becba5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:06.069375 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.069294 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhjj\" (UniqueName: \"kubernetes.io/projected/9de1d577-2e4c-4d73-bc42-eca44becba5e-kube-api-access-jrhjj\") pod \"payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn\" (UID: \"9de1d577-2e4c-4d73-bc42-eca44becba5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:06.069940 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.069918 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a66df6-d758-46a6-af02-909873244fca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v\" (UID: \"a8a66df6-d758-46a6-af02-909873244fca\") " pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:06.070048 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.070031 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de1d577-2e4c-4d73-bc42-eca44becba5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn\" (UID: \"9de1d577-2e4c-4d73-bc42-eca44becba5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:06.099073 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.099050 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhjj\" (UniqueName: \"kubernetes.io/projected/9de1d577-2e4c-4d73-bc42-eca44becba5e-kube-api-access-jrhjj\") pod \"payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn\" (UID: \"9de1d577-2e4c-4d73-bc42-eca44becba5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:06.101237 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.101212 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52dd9\" (UniqueName: \"kubernetes.io/projected/a8a66df6-d758-46a6-af02-909873244fca-kube-api-access-52dd9\") pod \"payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v\" (UID: \"a8a66df6-d758-46a6-af02-909873244fca\") " pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:06.192402 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.192354 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:06.213908 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.213883 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:06.364632 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.364599 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn"] Jul 24 13:31:06.367761 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:31:06.367732 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de1d577_2e4c_4d73_bc42_eca44becba5e.slice/crio-7f4961c67b86b52adf6cd05f724b22438573c0001762904b6a13f2ecf03aad4d WatchSource:0}: Error finding container 7f4961c67b86b52adf6cd05f724b22438573c0001762904b6a13f2ecf03aad4d: Status 404 returned error can't find the container with id 7f4961c67b86b52adf6cd05f724b22438573c0001762904b6a13f2ecf03aad4d Jul 24 13:31:06.382761 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.382718 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v"] Jul 24 13:31:06.386722 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:31:06.386697 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a66df6_d758_46a6_af02_909873244fca.slice/crio-d75afa4cf5f3ac3717b9712c83a16cc2422b6f2f292ff7e9321d0ddcbeac27a2 WatchSource:0}: Error finding container d75afa4cf5f3ac3717b9712c83a16cc2422b6f2f292ff7e9321d0ddcbeac27a2: Status 404 returned error can't find the container with id d75afa4cf5f3ac3717b9712c83a16cc2422b6f2f292ff7e9321d0ddcbeac27a2 Jul 24 13:31:06.874578 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.874523 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" event={"ID":"9de1d577-2e4c-4d73-bc42-eca44becba5e","Type":"ContainerStarted","Data":"71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea"} Jul 24 13:31:06.875039 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.874627 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:06.875039 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.874653 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" event={"ID":"9de1d577-2e4c-4d73-bc42-eca44becba5e","Type":"ContainerStarted","Data":"7f4961c67b86b52adf6cd05f724b22438573c0001762904b6a13f2ecf03aad4d"} Jul 24 13:31:06.876000 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.875976 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" event={"ID":"a8a66df6-d758-46a6-af02-909873244fca","Type":"ContainerStarted","Data":"98372e81d450b8a744875a215efdca45cf79cc6e6faa7ef51c6539269d02b594"} Jul 24 13:31:06.876124 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.876015 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" event={"ID":"a8a66df6-d758-46a6-af02-909873244fca","Type":"ContainerStarted","Data":"d75afa4cf5f3ac3717b9712c83a16cc2422b6f2f292ff7e9321d0ddcbeac27a2"} Jul 24 13:31:06.876124 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.876048 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:06.895284 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.895230 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" podStartSLOduration=1.895213929 podStartE2EDuration="1.895213929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:31:06.89391314 +0000 UTC m=+2282.897215243" watchObservedRunningTime="2026-07-24 13:31:06.895213929 +0000 UTC m=+2282.898516033" Jul 24 13:31:06.916399 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:06.916352 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" podStartSLOduration=1.916333051 podStartE2EDuration="1.916333051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:31:06.915085152 +0000 UTC m=+2282.918387259" watchObservedRunningTime="2026-07-24 13:31:06.916333051 +0000 UTC m=+2282.919635151" Jul 24 13:31:07.658148 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:07.658116 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:31:09.416712 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:09.416659 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:09.892127 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:09.892094 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:10.235926 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:10.235783 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:10.596180 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:10.596151 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:12.876955 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:12.876916 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:12.992364 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:12.992328 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:13.092106 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.092071 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:13.594239 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.594205 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:13.698356 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.698296 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:13.698841 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.698822 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff"] Jul 24 13:31:13.701845 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.701831 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:13.704479 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.704456 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a9463a\"" Jul 24 13:31:13.711985 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.711963 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff"] Jul 24 13:31:13.720120 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.720099 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9"] Jul 24 13:31:13.723174 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.723149 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:13.726254 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.726235 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a9463a-dockercfg-v8vbl\"" Jul 24 13:31:13.736741 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.736722 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9"] Jul 24 13:31:13.842370 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.842345 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac341d5e-0527-41e4-9741-615fa5369a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff\" (UID: \"ac341d5e-0527-41e4-9741-615fa5369a53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:13.842534 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.842379 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc95e92-89a0-43bf-9778-b579596bf286-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9\" (UID: \"5dc95e92-89a0-43bf-9778-b579596bf286\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:13.842534 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.842414 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcn56\" (UniqueName: \"kubernetes.io/projected/5dc95e92-89a0-43bf-9778-b579596bf286-kube-api-access-jcn56\") pod \"payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9\" (UID: \"5dc95e92-89a0-43bf-9778-b579596bf286\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:13.842534 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.842469 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwb4\" (UniqueName: \"kubernetes.io/projected/ac341d5e-0527-41e4-9741-615fa5369a53-kube-api-access-wbwb4\") pod \"payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff\" (UID: \"ac341d5e-0527-41e4-9741-615fa5369a53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:13.943067 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.942972 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac341d5e-0527-41e4-9741-615fa5369a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff\" (UID: \"ac341d5e-0527-41e4-9741-615fa5369a53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:13.943067 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.943018 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc95e92-89a0-43bf-9778-b579596bf286-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9\" (UID: \"5dc95e92-89a0-43bf-9778-b579596bf286\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:13.943067 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.943056 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcn56\" (UniqueName: \"kubernetes.io/projected/5dc95e92-89a0-43bf-9778-b579596bf286-kube-api-access-jcn56\") pod \"payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9\" (UID: \"5dc95e92-89a0-43bf-9778-b579596bf286\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:13.943542 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.943088 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwb4\" (UniqueName: \"kubernetes.io/projected/ac341d5e-0527-41e4-9741-615fa5369a53-kube-api-access-wbwb4\") pod \"payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff\" (UID: \"ac341d5e-0527-41e4-9741-615fa5369a53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:13.943795 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.943775 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac341d5e-0527-41e4-9741-615fa5369a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff\" (UID: \"ac341d5e-0527-41e4-9741-615fa5369a53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:13.943866 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.943848 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc95e92-89a0-43bf-9778-b579596bf286-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9\" (UID: \"5dc95e92-89a0-43bf-9778-b579596bf286\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:13.952299 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.952275 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwb4\" (UniqueName: \"kubernetes.io/projected/ac341d5e-0527-41e4-9741-615fa5369a53-kube-api-access-wbwb4\") pod \"payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff\" (UID: \"ac341d5e-0527-41e4-9741-615fa5369a53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:13.952641 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:13.952622 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcn56\" (UniqueName: \"kubernetes.io/projected/5dc95e92-89a0-43bf-9778-b579596bf286-kube-api-access-jcn56\") pod \"payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9\" (UID: \"5dc95e92-89a0-43bf-9778-b579596bf286\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:14.013630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.013591 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:14.033880 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.033853 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:14.173859 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.173836 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff"] Jul 24 13:31:14.176345 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:31:14.176320 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac341d5e_0527_41e4_9741_615fa5369a53.slice/crio-0a7f438590822605418c2ce76292e36cc962c703da009d5c5ddc0194d1ca62f5 WatchSource:0}: Error finding container 0a7f438590822605418c2ce76292e36cc962c703da009d5c5ddc0194d1ca62f5: Status 404 returned error can't find the container with id 0a7f438590822605418c2ce76292e36cc962c703da009d5c5ddc0194d1ca62f5 Jul 24 13:31:14.222320 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.222298 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9"] Jul 24 13:31:14.224175 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:31:14.224149 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc95e92_89a0_43bf_9778_b579596bf286.slice/crio-0a205bde0f811d355d0abeca9f9fcadd807b39d53d9107b8dd19a7005cfe781a WatchSource:0}: Error finding container 0a205bde0f811d355d0abeca9f9fcadd807b39d53d9107b8dd19a7005cfe781a: Status 404 returned error can't find the container with id 0a205bde0f811d355d0abeca9f9fcadd807b39d53d9107b8dd19a7005cfe781a Jul 24 13:31:14.910338 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.910295 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" event={"ID":"ac341d5e-0527-41e4-9741-615fa5369a53","Type":"ContainerStarted","Data":"e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb"} Jul 24 13:31:14.910537 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.910368 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:14.910537 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.910394 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" event={"ID":"ac341d5e-0527-41e4-9741-615fa5369a53","Type":"ContainerStarted","Data":"0a7f438590822605418c2ce76292e36cc962c703da009d5c5ddc0194d1ca62f5"} Jul 24 13:31:14.911826 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.911804 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" event={"ID":"5dc95e92-89a0-43bf-9778-b579596bf286","Type":"ContainerStarted","Data":"9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73"} Jul 24 13:31:14.911945 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.911840 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" event={"ID":"5dc95e92-89a0-43bf-9778-b579596bf286","Type":"ContainerStarted","Data":"0a205bde0f811d355d0abeca9f9fcadd807b39d53d9107b8dd19a7005cfe781a"} Jul 24 13:31:14.911945 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.911912 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:14.932082 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.932036 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" podStartSLOduration=1.9320217450000001 podStartE2EDuration="1.932021745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:31:14.929752006 +0000 UTC m=+2290.933054136" watchObservedRunningTime="2026-07-24 13:31:14.932021745 +0000 UTC m=+2290.935323849" Jul 24 13:31:14.947547 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:14.947503 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" podStartSLOduration=1.9474877560000001 podStartE2EDuration="1.947487756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:31:14.946467114 +0000 UTC m=+2290.949769245" watchObservedRunningTime="2026-07-24 13:31:14.947487756 +0000 UTC m=+2290.950789860" Jul 24 13:31:17.397330 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:17.397298 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:17.881974 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:17.881946 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:17.882129 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:17.882023 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:25.918731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:25.918695 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:31:25.919219 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:25.918771 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:31:26.663088 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:26.663057 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:31:41.890900 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:41.890866 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:41.908687 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:41.908652 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v"] Jul 24 13:31:41.908936 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:41.908896 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" podUID="a8a66df6-d758-46a6-af02-909873244fca" containerName="payload-processing" containerID="cri-o://98372e81d450b8a744875a215efdca45cf79cc6e6faa7ef51c6539269d02b594" gracePeriod=30 Jul 24 13:31:41.917033 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:41.917001 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn"] Jul 24 13:31:41.917274 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:41.917250 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" podUID="9de1d577-2e4c-4d73-bc42-eca44becba5e" containerName="payload-pre-processing" containerID="cri-o://71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea" gracePeriod=30 Jul 24 13:31:41.991335 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:41.991306 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:42.030367 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.030342 2554 generic.go:356] "Generic (PLEG): container finished" podID="a8a66df6-d758-46a6-af02-909873244fca" containerID="98372e81d450b8a744875a215efdca45cf79cc6e6faa7ef51c6539269d02b594" exitCode=0 Jul 24 13:31:42.030429 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.030394 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" event={"ID":"a8a66df6-d758-46a6-af02-909873244fca","Type":"ContainerDied","Data":"98372e81d450b8a744875a215efdca45cf79cc6e6faa7ef51c6539269d02b594"} Jul 24 13:31:42.073058 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.073025 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:42.182722 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.182701 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:42.185517 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.185499 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:42.283725 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.283672 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52dd9\" (UniqueName: \"kubernetes.io/projected/a8a66df6-d758-46a6-af02-909873244fca-kube-api-access-52dd9\") pod \"a8a66df6-d758-46a6-af02-909873244fca\" (UID: \"a8a66df6-d758-46a6-af02-909873244fca\") " Jul 24 13:31:42.283725 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.283728 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de1d577-2e4c-4d73-bc42-eca44becba5e-plugins-config-volume\") pod \"9de1d577-2e4c-4d73-bc42-eca44becba5e\" (UID: \"9de1d577-2e4c-4d73-bc42-eca44becba5e\") " Jul 24 13:31:42.283997 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.283761 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrhjj\" (UniqueName: \"kubernetes.io/projected/9de1d577-2e4c-4d73-bc42-eca44becba5e-kube-api-access-jrhjj\") pod \"9de1d577-2e4c-4d73-bc42-eca44becba5e\" (UID: \"9de1d577-2e4c-4d73-bc42-eca44becba5e\") " Jul 24 13:31:42.283997 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.283785 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a66df6-d758-46a6-af02-909873244fca-plugins-config-volume\") pod \"a8a66df6-d758-46a6-af02-909873244fca\" (UID: \"a8a66df6-d758-46a6-af02-909873244fca\") " Jul 24 13:31:42.284191 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.284166 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de1d577-2e4c-4d73-bc42-eca44becba5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9de1d577-2e4c-4d73-bc42-eca44becba5e" (UID: "9de1d577-2e4c-4d73-bc42-eca44becba5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:31:42.284191 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.284175 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a66df6-d758-46a6-af02-909873244fca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8a66df6-d758-46a6-af02-909873244fca" (UID: "a8a66df6-d758-46a6-af02-909873244fca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:31:42.286010 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.285988 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a66df6-d758-46a6-af02-909873244fca-kube-api-access-52dd9" (OuterVolumeSpecName: "kube-api-access-52dd9") pod "a8a66df6-d758-46a6-af02-909873244fca" (UID: "a8a66df6-d758-46a6-af02-909873244fca"). InnerVolumeSpecName "kube-api-access-52dd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:31:42.286010 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.286002 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de1d577-2e4c-4d73-bc42-eca44becba5e-kube-api-access-jrhjj" (OuterVolumeSpecName: "kube-api-access-jrhjj") pod "9de1d577-2e4c-4d73-bc42-eca44becba5e" (UID: "9de1d577-2e4c-4d73-bc42-eca44becba5e"). InnerVolumeSpecName "kube-api-access-jrhjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:31:42.384693 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.384651 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52dd9\" (UniqueName: \"kubernetes.io/projected/a8a66df6-d758-46a6-af02-909873244fca-kube-api-access-52dd9\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:31:42.384693 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.384690 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de1d577-2e4c-4d73-bc42-eca44becba5e-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:31:42.384693 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.384700 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrhjj\" (UniqueName: \"kubernetes.io/projected/9de1d577-2e4c-4d73-bc42-eca44becba5e-kube-api-access-jrhjj\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:31:42.384693 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:42.384709 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a66df6-d758-46a6-af02-909873244fca-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:31:43.036307 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.036267 2554 generic.go:356] "Generic (PLEG): container finished" podID="9de1d577-2e4c-4d73-bc42-eca44becba5e" containerID="71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea" exitCode=0 Jul 24 13:31:43.036766 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.036317 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" Jul 24 13:31:43.036766 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.036348 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" event={"ID":"9de1d577-2e4c-4d73-bc42-eca44becba5e","Type":"ContainerDied","Data":"71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea"} Jul 24 13:31:43.036766 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.036400 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn" event={"ID":"9de1d577-2e4c-4d73-bc42-eca44becba5e","Type":"ContainerDied","Data":"7f4961c67b86b52adf6cd05f724b22438573c0001762904b6a13f2ecf03aad4d"} Jul 24 13:31:43.036766 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.036425 2554 scope.go:117] "RemoveContainer" containerID="71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea" Jul 24 13:31:43.037775 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.037748 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" Jul 24 13:31:43.037775 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.037763 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v" event={"ID":"a8a66df6-d758-46a6-af02-909873244fca","Type":"ContainerDied","Data":"d75afa4cf5f3ac3717b9712c83a16cc2422b6f2f292ff7e9321d0ddcbeac27a2"} Jul 24 13:31:43.044874 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.044856 2554 scope.go:117] "RemoveContainer" containerID="71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea" Jul 24 13:31:43.045132 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:31:43.045116 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea\": container with ID starting with 71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea not found: ID does not exist" containerID="71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea" Jul 24 13:31:43.045185 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.045140 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea"} err="failed to get container status \"71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea\": rpc error: code = NotFound desc = could not find container \"71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea\": container with ID starting with 71e46a1d816bc6ad7c370b245a0514b8a8ca6269a6cea444f231068b248461ea not found: ID does not exist" Jul 24 13:31:43.045185 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.045155 2554 scope.go:117] "RemoveContainer" containerID="98372e81d450b8a744875a215efdca45cf79cc6e6faa7ef51c6539269d02b594" Jul 24 13:31:43.064773 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.064747 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn"] Jul 24 13:31:43.070574 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.070537 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e530f2-866c776488-4dgcn"] Jul 24 13:31:43.088323 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.088289 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v"] Jul 24 13:31:43.094196 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.094173 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e530f2-6c66d88cb6-zcv4v"] Jul 24 13:31:43.126286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:43.126255 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:44.598832 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:44.598788 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de1d577-2e4c-4d73-bc42-eca44becba5e" path="/var/lib/kubelet/pods/9de1d577-2e4c-4d73-bc42-eca44becba5e/volumes" Jul 24 13:31:44.599323 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:44.599280 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8a66df6-d758-46a6-af02-909873244fca" path="/var/lib/kubelet/pods/a8a66df6-d758-46a6-af02-909873244fca/volumes" Jul 24 13:31:45.455682 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:45.455649 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:31:45.701532 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:45.701505 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:46.085711 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:46.085671 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:52.501613 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:52.501551 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:52.597087 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:52.597057 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:52.795549 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:52.795521 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:31:53.032206 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:31:53.032173 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:04.685983 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:04.685908 2554 scope.go:117] "RemoveContainer" containerID="7bdc5e5d37f096d50d1f1ec7749e2b49d1c150b86f275a31815c284eb7a56e8f" Jul 24 13:32:08.733939 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:08.733906 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:08.753650 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:08.753623 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9"] Jul 24 13:32:08.753873 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:08.753852 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" podUID="5dc95e92-89a0-43bf-9778-b579596bf286" containerName="payload-processing" containerID="cri-o://9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73" gracePeriod=30 Jul 24 13:32:08.759810 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:08.759769 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff"] Jul 24 13:32:08.759996 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:08.759971 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" podUID="ac341d5e-0527-41e4-9741-615fa5369a53" containerName="payload-pre-processing" containerID="cri-o://e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb" gracePeriod=30 Jul 24 13:32:08.831020 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:08.830997 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:09.026612 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.026588 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:32:09.029207 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.029189 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:32:09.030675 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.030655 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbwb4\" (UniqueName: \"kubernetes.io/projected/ac341d5e-0527-41e4-9741-615fa5369a53-kube-api-access-wbwb4\") pod \"ac341d5e-0527-41e4-9741-615fa5369a53\" (UID: \"ac341d5e-0527-41e4-9741-615fa5369a53\") " Jul 24 13:32:09.030786 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.030686 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc95e92-89a0-43bf-9778-b579596bf286-plugins-config-volume\") pod \"5dc95e92-89a0-43bf-9778-b579596bf286\" (UID: \"5dc95e92-89a0-43bf-9778-b579596bf286\") " Jul 24 13:32:09.030786 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.030708 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcn56\" (UniqueName: \"kubernetes.io/projected/5dc95e92-89a0-43bf-9778-b579596bf286-kube-api-access-jcn56\") pod \"5dc95e92-89a0-43bf-9778-b579596bf286\" (UID: \"5dc95e92-89a0-43bf-9778-b579596bf286\") " Jul 24 13:32:09.030786 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.030748 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac341d5e-0527-41e4-9741-615fa5369a53-plugins-config-volume\") pod \"ac341d5e-0527-41e4-9741-615fa5369a53\" (UID: \"ac341d5e-0527-41e4-9741-615fa5369a53\") " Jul 24 13:32:09.031088 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.031058 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc95e92-89a0-43bf-9778-b579596bf286-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5dc95e92-89a0-43bf-9778-b579596bf286" (UID: "5dc95e92-89a0-43bf-9778-b579596bf286"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:32:09.031198 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.031129 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac341d5e-0527-41e4-9741-615fa5369a53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac341d5e-0527-41e4-9741-615fa5369a53" (UID: "ac341d5e-0527-41e4-9741-615fa5369a53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:32:09.032837 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.032813 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac341d5e-0527-41e4-9741-615fa5369a53-kube-api-access-wbwb4" (OuterVolumeSpecName: "kube-api-access-wbwb4") pod "ac341d5e-0527-41e4-9741-615fa5369a53" (UID: "ac341d5e-0527-41e4-9741-615fa5369a53"). InnerVolumeSpecName "kube-api-access-wbwb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:32:09.032933 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.032892 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc95e92-89a0-43bf-9778-b579596bf286-kube-api-access-jcn56" (OuterVolumeSpecName: "kube-api-access-jcn56") pod "5dc95e92-89a0-43bf-9778-b579596bf286" (UID: "5dc95e92-89a0-43bf-9778-b579596bf286"). InnerVolumeSpecName "kube-api-access-jcn56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:32:09.132131 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.132094 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbwb4\" (UniqueName: \"kubernetes.io/projected/ac341d5e-0527-41e4-9741-615fa5369a53-kube-api-access-wbwb4\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:32:09.132131 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.132124 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc95e92-89a0-43bf-9778-b579596bf286-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:32:09.132131 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.132135 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcn56\" (UniqueName: \"kubernetes.io/projected/5dc95e92-89a0-43bf-9778-b579596bf286-kube-api-access-jcn56\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:32:09.132373 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.132144 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac341d5e-0527-41e4-9741-615fa5369a53-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:32:09.145678 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.145649 2554 generic.go:356] "Generic (PLEG): container finished" podID="ac341d5e-0527-41e4-9741-615fa5369a53" containerID="e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb" exitCode=0 Jul 24 13:32:09.145852 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.145694 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" Jul 24 13:32:09.145852 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.145747 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" event={"ID":"ac341d5e-0527-41e4-9741-615fa5369a53","Type":"ContainerDied","Data":"e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb"} Jul 24 13:32:09.145852 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.145815 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff" event={"ID":"ac341d5e-0527-41e4-9741-615fa5369a53","Type":"ContainerDied","Data":"0a7f438590822605418c2ce76292e36cc962c703da009d5c5ddc0194d1ca62f5"} Jul 24 13:32:09.145852 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.145846 2554 scope.go:117] "RemoveContainer" containerID="e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb" Jul 24 13:32:09.147061 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.147040 2554 generic.go:356] "Generic (PLEG): container finished" podID="5dc95e92-89a0-43bf-9778-b579596bf286" containerID="9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73" exitCode=0 Jul 24 13:32:09.147150 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.147081 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" Jul 24 13:32:09.147150 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.147108 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" event={"ID":"5dc95e92-89a0-43bf-9778-b579596bf286","Type":"ContainerDied","Data":"9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73"} Jul 24 13:32:09.147255 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.147150 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9" event={"ID":"5dc95e92-89a0-43bf-9778-b579596bf286","Type":"ContainerDied","Data":"0a205bde0f811d355d0abeca9f9fcadd807b39d53d9107b8dd19a7005cfe781a"} Jul 24 13:32:09.154127 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.154090 2554 scope.go:117] "RemoveContainer" containerID="e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb" Jul 24 13:32:09.154505 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:32:09.154481 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb\": container with ID starting with e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb not found: ID does not exist" containerID="e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb" Jul 24 13:32:09.154627 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.154514 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb"} err="failed to get container status \"e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb\": rpc error: code = NotFound desc = could not find container \"e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb\": container with ID starting with e618163a569d03b1d786e57b84a22f1e6d3decdb6441fa9e3a2f5c9944da4cfb not found: ID does not exist" Jul 24 13:32:09.154627 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.154536 2554 scope.go:117] "RemoveContainer" containerID="9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73" Jul 24 13:32:09.162637 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.162613 2554 scope.go:117] "RemoveContainer" containerID="9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73" Jul 24 13:32:09.162868 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:32:09.162853 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73\": container with ID starting with 9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73 not found: ID does not exist" containerID="9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73" Jul 24 13:32:09.162913 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.162875 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73"} err="failed to get container status \"9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73\": rpc error: code = NotFound desc = could not find container \"9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73\": container with ID starting with 9e0cc6c8018457cc6b5bb899ae38e9c65f3caf80cfe7b2c78d63092a53d2ab73 not found: ID does not exist" Jul 24 13:32:09.172262 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.172243 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff"] Jul 24 13:32:09.178283 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.178262 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9463a-5fd75b4484-mm7ff"] Jul 24 13:32:09.191609 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.191587 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9"] Jul 24 13:32:09.196413 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:09.196392 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9463a-5cb595489d-d82r9"] Jul 24 13:32:10.598081 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:10.598050 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc95e92-89a0-43bf-9778-b579596bf286" path="/var/lib/kubelet/pods/5dc95e92-89a0-43bf-9778-b579596bf286/volumes" Jul 24 13:32:10.598471 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:10.598382 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac341d5e-0527-41e4-9741-615fa5369a53" path="/var/lib/kubelet/pods/ac341d5e-0527-41e4-9741-615fa5369a53/volumes" Jul 24 13:32:12.854187 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:12.854150 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:32:13.256102 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:13.256008 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:13.677174 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:13.677134 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:21.029058 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:21.029023 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:21.133852 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:21.133820 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:21.320629 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:21.320593 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:21.598482 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:21.598389 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:21.924129 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:21.924037 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:22.295079 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:22.295052 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:22.610581 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:22.610475 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:25.303349 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:25.303318 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:25.431705 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:25.431677 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:25.529751 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:25.529706 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:25.935780 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:25.935744 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:26.141771 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.141727 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:26.262922 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.262842 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:26.361475 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361442 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5"] Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361775 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de1d577-2e4c-4d73-bc42-eca44becba5e" containerName="payload-pre-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361787 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de1d577-2e4c-4d73-bc42-eca44becba5e" containerName="payload-pre-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361797 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac341d5e-0527-41e4-9741-615fa5369a53" containerName="payload-pre-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361802 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac341d5e-0527-41e4-9741-615fa5369a53" containerName="payload-pre-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361811 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc95e92-89a0-43bf-9778-b579596bf286" containerName="payload-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361817 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc95e92-89a0-43bf-9778-b579596bf286" containerName="payload-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361830 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a66df6-d758-46a6-af02-909873244fca" containerName="payload-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361836 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a66df6-d758-46a6-af02-909873244fca" containerName="payload-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361897 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a66df6-d758-46a6-af02-909873244fca" containerName="payload-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361905 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac341d5e-0527-41e4-9741-615fa5369a53" containerName="payload-pre-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361912 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de1d577-2e4c-4d73-bc42-eca44becba5e" containerName="payload-pre-processing" Jul 24 13:32:26.361910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.361918 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc95e92-89a0-43bf-9778-b579596bf286" containerName="payload-processing" Jul 24 13:32:26.364701 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.364686 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:26.367544 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.367518 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-f3a913\"" Jul 24 13:32:26.367721 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.367706 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-f3a913-dockercfg-pqfj7\"" Jul 24 13:32:26.377879 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.377858 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5"] Jul 24 13:32:26.443052 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.443023 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2"] Jul 24 13:32:26.446104 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.446088 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:26.470975 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.470942 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2"] Jul 24 13:32:26.475459 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.475435 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7qqw\" (UniqueName: \"kubernetes.io/projected/02063e5e-9e36-4141-9b29-a4c334b8f007-kube-api-access-m7qqw\") pod \"payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2\" (UID: \"02063e5e-9e36-4141-9b29-a4c334b8f007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:26.475633 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.475480 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02063e5e-9e36-4141-9b29-a4c334b8f007-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2\" (UID: \"02063e5e-9e36-4141-9b29-a4c334b8f007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:26.475633 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.475511 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8rr6\" (UniqueName: \"kubernetes.io/projected/8f053ba3-c11d-446c-a33f-70f714678f42-kube-api-access-t8rr6\") pod \"payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5\" (UID: \"8f053ba3-c11d-446c-a33f-70f714678f42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:26.475752 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.475659 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f053ba3-c11d-446c-a33f-70f714678f42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5\" (UID: \"8f053ba3-c11d-446c-a33f-70f714678f42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:26.576774 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.576739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02063e5e-9e36-4141-9b29-a4c334b8f007-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2\" (UID: \"02063e5e-9e36-4141-9b29-a4c334b8f007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:26.576958 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.576788 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8rr6\" (UniqueName: \"kubernetes.io/projected/8f053ba3-c11d-446c-a33f-70f714678f42-kube-api-access-t8rr6\") pod \"payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5\" (UID: \"8f053ba3-c11d-446c-a33f-70f714678f42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:26.576958 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.576861 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f053ba3-c11d-446c-a33f-70f714678f42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5\" (UID: \"8f053ba3-c11d-446c-a33f-70f714678f42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:26.576958 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.576897 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7qqw\" (UniqueName: \"kubernetes.io/projected/02063e5e-9e36-4141-9b29-a4c334b8f007-kube-api-access-m7qqw\") pod \"payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2\" (UID: \"02063e5e-9e36-4141-9b29-a4c334b8f007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:26.577587 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.577547 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02063e5e-9e36-4141-9b29-a4c334b8f007-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2\" (UID: \"02063e5e-9e36-4141-9b29-a4c334b8f007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:26.577630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.577600 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f053ba3-c11d-446c-a33f-70f714678f42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5\" (UID: \"8f053ba3-c11d-446c-a33f-70f714678f42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:26.590474 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.590448 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7qqw\" (UniqueName: \"kubernetes.io/projected/02063e5e-9e36-4141-9b29-a4c334b8f007-kube-api-access-m7qqw\") pod \"payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2\" (UID: \"02063e5e-9e36-4141-9b29-a4c334b8f007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:26.591823 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.591804 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8rr6\" (UniqueName: \"kubernetes.io/projected/8f053ba3-c11d-446c-a33f-70f714678f42-kube-api-access-t8rr6\") pod \"payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5\" (UID: \"8f053ba3-c11d-446c-a33f-70f714678f42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:26.674637 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.674609 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:26.756085 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.755800 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:26.813731 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.813700 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5"] Jul 24 13:32:26.815933 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:32:26.815905 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f053ba3_c11d_446c_a33f_70f714678f42.slice/crio-f1ec072cf7df42b2f398c1c2a7f1d127e6deecaaf6b90f57215e50f14f673e56 WatchSource:0}: Error finding container f1ec072cf7df42b2f398c1c2a7f1d127e6deecaaf6b90f57215e50f14f673e56: Status 404 returned error can't find the container with id f1ec072cf7df42b2f398c1c2a7f1d127e6deecaaf6b90f57215e50f14f673e56 Jul 24 13:32:26.895184 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:26.895051 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2"] Jul 24 13:32:26.897706 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:32:26.897678 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02063e5e_9e36_4141_9b29_a4c334b8f007.slice/crio-b35699468fbaebe123dcf840ab029d0566270aaeaaaef61596b9a024e1cd19c2 WatchSource:0}: Error finding container b35699468fbaebe123dcf840ab029d0566270aaeaaaef61596b9a024e1cd19c2: Status 404 returned error can't find the container with id b35699468fbaebe123dcf840ab029d0566270aaeaaaef61596b9a024e1cd19c2 Jul 24 13:32:27.227521 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:27.227429 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" event={"ID":"02063e5e-9e36-4141-9b29-a4c334b8f007","Type":"ContainerStarted","Data":"e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1"} Jul 24 13:32:27.227521 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:27.227487 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" event={"ID":"02063e5e-9e36-4141-9b29-a4c334b8f007","Type":"ContainerStarted","Data":"b35699468fbaebe123dcf840ab029d0566270aaeaaaef61596b9a024e1cd19c2"} Jul 24 13:32:27.227794 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:27.227540 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:27.228922 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:27.228899 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" event={"ID":"8f053ba3-c11d-446c-a33f-70f714678f42","Type":"ContainerStarted","Data":"3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9"} Jul 24 13:32:27.229034 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:27.228936 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" event={"ID":"8f053ba3-c11d-446c-a33f-70f714678f42","Type":"ContainerStarted","Data":"f1ec072cf7df42b2f398c1c2a7f1d127e6deecaaf6b90f57215e50f14f673e56"} Jul 24 13:32:27.229034 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:27.228974 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:27.247008 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:27.246964 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" podStartSLOduration=1.246948045 podStartE2EDuration="1.246948045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:32:27.245925933 +0000 UTC m=+2363.249228038" watchObservedRunningTime="2026-07-24 13:32:27.246948045 +0000 UTC m=+2363.250250146" Jul 24 13:32:27.263621 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:27.263546 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" podStartSLOduration=1.263530679 podStartE2EDuration="1.263530679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:32:27.262392583 +0000 UTC m=+2363.265694687" watchObservedRunningTime="2026-07-24 13:32:27.263530679 +0000 UTC m=+2363.266832779" Jul 24 13:32:29.772820 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:29.772786 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:30.125928 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:30.125877 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:30.492888 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:30.492809 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:30.591581 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:30.591525 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:33.297050 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:33.297020 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:33.393353 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:33.393324 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:33.531739 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:33.531700 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:33.970775 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:33.970735 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:34.133094 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.133064 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:34.240425 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.240338 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:34.262795 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.262765 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z"] Jul 24 13:32:34.266199 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.266180 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt"] Jul 24 13:32:34.266316 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.266299 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:34.269015 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.268999 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:34.269240 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.269218 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-b1a3df\"" Jul 24 13:32:34.274882 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.274859 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:32:34.284352 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.284279 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z"] Jul 24 13:32:34.285986 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.285965 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt"] Jul 24 13:32:34.347226 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.347193 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d862a456-95ed-4f15-a6dc-73bee61f4f3f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt\" (UID: \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:34.347684 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.347246 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lrcl\" (UniqueName: \"kubernetes.io/projected/d862a456-95ed-4f15-a6dc-73bee61f4f3f-kube-api-access-8lrcl\") pod \"payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt\" (UID: \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:34.347684 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.347332 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d493ddf-88f1-4f51-9be5-da2d64442fcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z\" (UID: \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:34.347684 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.347365 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mll\" (UniqueName: \"kubernetes.io/projected/1d493ddf-88f1-4f51-9be5-da2d64442fcc-kube-api-access-g6mll\") pod \"payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z\" (UID: \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:34.448601 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.448571 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d493ddf-88f1-4f51-9be5-da2d64442fcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z\" (UID: \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:34.448748 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.448626 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mll\" (UniqueName: \"kubernetes.io/projected/1d493ddf-88f1-4f51-9be5-da2d64442fcc-kube-api-access-g6mll\") pod \"payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z\" (UID: \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:34.448748 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.448663 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d862a456-95ed-4f15-a6dc-73bee61f4f3f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt\" (UID: \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:34.448748 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.448702 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lrcl\" (UniqueName: \"kubernetes.io/projected/d862a456-95ed-4f15-a6dc-73bee61f4f3f-kube-api-access-8lrcl\") pod \"payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt\" (UID: \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:34.449206 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.449185 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d493ddf-88f1-4f51-9be5-da2d64442fcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z\" (UID: \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:34.449306 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.449288 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d862a456-95ed-4f15-a6dc-73bee61f4f3f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt\" (UID: \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:34.466555 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.466521 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mll\" (UniqueName: \"kubernetes.io/projected/1d493ddf-88f1-4f51-9be5-da2d64442fcc-kube-api-access-g6mll\") pod \"payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z\" (UID: \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:34.467789 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.467773 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lrcl\" (UniqueName: \"kubernetes.io/projected/d862a456-95ed-4f15-a6dc-73bee61f4f3f-kube-api-access-8lrcl\") pod \"payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt\" (UID: \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:34.577501 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.577468 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:34.584497 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.584475 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:34.743013 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.742987 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt"] Jul 24 13:32:34.745934 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:32:34.745902 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd862a456_95ed_4f15_a6dc_73bee61f4f3f.slice/crio-73ed45bb27e280fb896492cefe50185d5182a7651bee0383fdac00967dcf457a WatchSource:0}: Error finding container 73ed45bb27e280fb896492cefe50185d5182a7651bee0383fdac00967dcf457a: Status 404 returned error can't find the container with id 73ed45bb27e280fb896492cefe50185d5182a7651bee0383fdac00967dcf457a Jul 24 13:32:34.938163 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:34.933793 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z"] Jul 24 13:32:34.946867 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:32:34.946839 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d493ddf_88f1_4f51_9be5_da2d64442fcc.slice/crio-aff2deb14e646c8539d92bf51bc4bdc9eaeebe5d5943ac0e7035c1607ee10ee9 WatchSource:0}: Error finding container aff2deb14e646c8539d92bf51bc4bdc9eaeebe5d5943ac0e7035c1607ee10ee9: Status 404 returned error can't find the container with id aff2deb14e646c8539d92bf51bc4bdc9eaeebe5d5943ac0e7035c1607ee10ee9 Jul 24 13:32:35.200404 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.200316 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:35.265672 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.265633 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" event={"ID":"d862a456-95ed-4f15-a6dc-73bee61f4f3f","Type":"ContainerStarted","Data":"655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a"} Jul 24 13:32:35.265863 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.265702 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" event={"ID":"d862a456-95ed-4f15-a6dc-73bee61f4f3f","Type":"ContainerStarted","Data":"73ed45bb27e280fb896492cefe50185d5182a7651bee0383fdac00967dcf457a"} Jul 24 13:32:35.265863 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.265742 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:35.267068 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.267044 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" event={"ID":"1d493ddf-88f1-4f51-9be5-da2d64442fcc","Type":"ContainerStarted","Data":"991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc"} Jul 24 13:32:35.267176 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.267090 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" event={"ID":"1d493ddf-88f1-4f51-9be5-da2d64442fcc","Type":"ContainerStarted","Data":"aff2deb14e646c8539d92bf51bc4bdc9eaeebe5d5943ac0e7035c1607ee10ee9"} Jul 24 13:32:35.267176 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.267138 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:35.290753 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.290706 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" podStartSLOduration=1.290692939 podStartE2EDuration="1.290692939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:32:35.289068516 +0000 UTC m=+2371.292370620" watchObservedRunningTime="2026-07-24 13:32:35.290692939 +0000 UTC m=+2371.293995021" Jul 24 13:32:35.313863 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.313813 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" podStartSLOduration=1.313795267 podStartE2EDuration="1.313795267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:32:35.312088698 +0000 UTC m=+2371.315390814" watchObservedRunningTime="2026-07-24 13:32:35.313795267 +0000 UTC m=+2371.317097371" Jul 24 13:32:35.338301 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:35.338275 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:37.804356 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:37.804323 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:32:38.236018 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:38.235930 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:32:38.236018 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:38.236011 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:32:46.273092 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:46.273053 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:32:46.273553 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:46.273134 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:32:46.577246 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:46.577215 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:32:57.911004 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:32:57.910969 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:33:01.362169 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:01.362128 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:33:20.507603 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.507554 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz"] Jul 24 13:33:20.510392 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.510373 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.514346 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.514310 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"openshift-service-ca.crt\"" Jul 24 13:33:20.514346 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.514336 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"default-dockercfg-dk7vn\"" Jul 24 13:33:20.514603 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.514350 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"kube-root-ca.crt\"" Jul 24 13:33:20.514603 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.514334 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1a3df\"/\"auth-test-model-b1a3df-kserve-self-signed-certs\"" Jul 24 13:33:20.521507 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.521486 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz"] Jul 24 13:33:20.553956 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.553927 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:33:20.673688 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.673629 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-tmp-dir\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.673889 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.673754 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-dshm\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.673933 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.673889 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-home\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.673933 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.673923 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-model-cache\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.674003 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.673940 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae77048-0040-4110-924f-100e406cf4f8-tls-certs\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.674003 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.673994 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbkvb\" (UniqueName: \"kubernetes.io/projected/cae77048-0040-4110-924f-100e406cf4f8-kube-api-access-sbkvb\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.775553 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.775465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-tmp-dir\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.775553 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.775514 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-dshm\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.775751 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.775638 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-home\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.775751 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.775682 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-model-cache\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.775823 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.775792 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae77048-0040-4110-924f-100e406cf4f8-tls-certs\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.775867 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.775838 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbkvb\" (UniqueName: \"kubernetes.io/projected/cae77048-0040-4110-924f-100e406cf4f8-kube-api-access-sbkvb\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.775993 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.775973 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-tmp-dir\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.776066 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.776032 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-model-cache\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.776108 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.776065 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-home\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.777931 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.777908 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-dshm\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.778222 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.778204 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae77048-0040-4110-924f-100e406cf4f8-tls-certs\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.785965 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.785944 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbkvb\" (UniqueName: \"kubernetes.io/projected/cae77048-0040-4110-924f-100e406cf4f8-kube-api-access-sbkvb\") pod \"auth-test-model-b1a3df-kserve-578897b6c9-zftlz\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.819646 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.819611 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:20.954881 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:20.954849 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz"] Jul 24 13:33:20.958185 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:33:20.958157 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcae77048_0040_4110_924f_100e406cf4f8.slice/crio-e33f6efced305a3eedb07e145c6312cee16322492917b0cd938560c2503770f5 WatchSource:0}: Error finding container e33f6efced305a3eedb07e145c6312cee16322492917b0cd938560c2503770f5: Status 404 returned error can't find the container with id e33f6efced305a3eedb07e145c6312cee16322492917b0cd938560c2503770f5 Jul 24 13:33:21.456090 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:21.456058 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" event={"ID":"cae77048-0040-4110-924f-100e406cf4f8","Type":"ContainerStarted","Data":"9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177"} Jul 24 13:33:21.456090 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:21.456097 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" event={"ID":"cae77048-0040-4110-924f-100e406cf4f8","Type":"ContainerStarted","Data":"e33f6efced305a3eedb07e145c6312cee16322492917b0cd938560c2503770f5"} Jul 24 13:33:21.478254 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:21.478201 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" podStartSLOduration=1.4781866209999999 podStartE2EDuration="1.478186621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:33:21.476829924 +0000 UTC m=+2417.480132029" watchObservedRunningTime="2026-07-24 13:33:21.478186621 +0000 UTC m=+2417.481488726" Jul 24 13:33:24.164574 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:24.164532 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:33:30.820712 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:30.820676 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:30.823090 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:30.820734 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:30.833513 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:30.833488 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:31.508294 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:31.508262 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:33:41.863461 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:41.863424 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:33:55.566083 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:33:55.566042 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:34:11.950658 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:34:11.950614 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:34:28.154716 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:34:28.154681 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:34:44.264754 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:34:44.264711 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:35:00.161418 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:00.161335 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:35:12.557303 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:12.557272 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:35:13.341622 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:13.341590 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz"] Jul 24 13:35:13.370284 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:13.370256 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:35:15.308614 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.308579 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l"] Jul 24 13:35:15.312160 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.312136 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.316129 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.316100 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f3a913\"/\"kube-root-ca.crt\"" Jul 24 13:35:15.316286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.316139 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f3a913\"/\"openshift-service-ca.crt\"" Jul 24 13:35:15.316286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.316139 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f3a913\"/\"sub-test-model-f3a913-kserve-self-signed-certs\"" Jul 24 13:35:15.316286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.316100 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f3a913\"/\"default-dockercfg-psg89\"" Jul 24 13:35:15.322079 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.322058 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l"] Jul 24 13:35:15.345155 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.345127 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:35:15.390630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.390605 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-dshm\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.390747 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.390642 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppmr4\" (UniqueName: \"kubernetes.io/projected/0ca45f63-da56-48b0-839f-9a2e279842c1-kube-api-access-ppmr4\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.390747 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.390709 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca45f63-da56-48b0-839f-9a2e279842c1-tls-certs\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.390828 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.390760 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-tmp-dir\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.390828 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.390782 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-home\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.390828 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.390806 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-model-cache\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.491321 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.491287 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-tmp-dir\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.491321 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.491325 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-home\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.491536 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.491346 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-model-cache\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.491536 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.491384 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-dshm\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.491536 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.491417 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppmr4\" (UniqueName: \"kubernetes.io/projected/0ca45f63-da56-48b0-839f-9a2e279842c1-kube-api-access-ppmr4\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.491536 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.491459 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca45f63-da56-48b0-839f-9a2e279842c1-tls-certs\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.491786 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.491763 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-tmp-dir\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.491845 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.491791 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-model-cache\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.491884 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.491836 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-home\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.493933 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.493909 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-dshm\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.494034 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.493983 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca45f63-da56-48b0-839f-9a2e279842c1-tls-certs\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.501624 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.501601 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppmr4\" (UniqueName: \"kubernetes.io/projected/0ca45f63-da56-48b0-839f-9a2e279842c1-kube-api-access-ppmr4\") pod \"sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.624241 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.624154 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:15.754395 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.754368 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l"] Jul 24 13:35:15.756202 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:35:15.756173 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ca45f63_da56_48b0_839f_9a2e279842c1.slice/crio-1f5f47da41cd90adc597e6b017e73b8ba9d880d20f28193c5100314253fc8bae WatchSource:0}: Error finding container 1f5f47da41cd90adc597e6b017e73b8ba9d880d20f28193c5100314253fc8bae: Status 404 returned error can't find the container with id 1f5f47da41cd90adc597e6b017e73b8ba9d880d20f28193c5100314253fc8bae Jul 24 13:35:15.944999 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.944898 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" event={"ID":"0ca45f63-da56-48b0-839f-9a2e279842c1","Type":"ContainerStarted","Data":"04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f"} Jul 24 13:35:15.944999 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.944941 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" event={"ID":"0ca45f63-da56-48b0-839f-9a2e279842c1","Type":"ContainerStarted","Data":"1f5f47da41cd90adc597e6b017e73b8ba9d880d20f28193c5100314253fc8bae"} Jul 24 13:35:15.963185 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:15.963137 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" podStartSLOduration=0.963121625 podStartE2EDuration="963.121625ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:15.96267315 +0000 UTC m=+2531.965975275" watchObservedRunningTime="2026-07-24 13:35:15.963121625 +0000 UTC m=+2531.966423728" Jul 24 13:35:18.867476 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:18.867439 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:35:25.624446 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:25.624406 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:25.624446 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:25.624457 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:25.637248 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:25.637226 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:26.000553 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:26.000454 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:35:28.360787 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.360726 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" podUID="cae77048-0040-4110-924f-100e406cf4f8" containerName="main" containerID="cri-o://9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177" gracePeriod=45 Jul 24 13:35:28.609209 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.609184 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:35:28.694723 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.694646 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-model-cache\") pod \"cae77048-0040-4110-924f-100e406cf4f8\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " Jul 24 13:35:28.694723 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.694681 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae77048-0040-4110-924f-100e406cf4f8-tls-certs\") pod \"cae77048-0040-4110-924f-100e406cf4f8\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " Jul 24 13:35:28.694924 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.694723 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-home\") pod \"cae77048-0040-4110-924f-100e406cf4f8\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " Jul 24 13:35:28.694924 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.694784 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbkvb\" (UniqueName: \"kubernetes.io/projected/cae77048-0040-4110-924f-100e406cf4f8-kube-api-access-sbkvb\") pod \"cae77048-0040-4110-924f-100e406cf4f8\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " Jul 24 13:35:28.694924 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.694831 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-dshm\") pod \"cae77048-0040-4110-924f-100e406cf4f8\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " Jul 24 13:35:28.694924 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.694887 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-tmp-dir\") pod \"cae77048-0040-4110-924f-100e406cf4f8\" (UID: \"cae77048-0040-4110-924f-100e406cf4f8\") " Jul 24 13:35:28.695132 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.694951 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "cae77048-0040-4110-924f-100e406cf4f8" (UID: "cae77048-0040-4110-924f-100e406cf4f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:35:28.695132 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.694981 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-home" (OuterVolumeSpecName: "home") pod "cae77048-0040-4110-924f-100e406cf4f8" (UID: "cae77048-0040-4110-924f-100e406cf4f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:35:28.695249 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.695182 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cae77048-0040-4110-924f-100e406cf4f8" (UID: "cae77048-0040-4110-924f-100e406cf4f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:35:28.695249 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.695212 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-model-cache\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:35:28.695249 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.695232 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-home\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:35:28.697178 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.697148 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae77048-0040-4110-924f-100e406cf4f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cae77048-0040-4110-924f-100e406cf4f8" (UID: "cae77048-0040-4110-924f-100e406cf4f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:35:28.697178 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.697163 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-dshm" (OuterVolumeSpecName: "dshm") pod "cae77048-0040-4110-924f-100e406cf4f8" (UID: "cae77048-0040-4110-924f-100e406cf4f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:35:28.697325 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.697179 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae77048-0040-4110-924f-100e406cf4f8-kube-api-access-sbkvb" (OuterVolumeSpecName: "kube-api-access-sbkvb") pod "cae77048-0040-4110-924f-100e406cf4f8" (UID: "cae77048-0040-4110-924f-100e406cf4f8"). InnerVolumeSpecName "kube-api-access-sbkvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:35:28.795927 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.795887 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-tmp-dir\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:35:28.795927 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.795921 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae77048-0040-4110-924f-100e406cf4f8-tls-certs\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:35:28.795927 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.795930 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbkvb\" (UniqueName: \"kubernetes.io/projected/cae77048-0040-4110-924f-100e406cf4f8-kube-api-access-sbkvb\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:35:28.795927 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:28.795938 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae77048-0040-4110-924f-100e406cf4f8-dshm\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:35:29.001634 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:29.001530 2554 generic.go:356] "Generic (PLEG): container finished" podID="cae77048-0040-4110-924f-100e406cf4f8" containerID="9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177" exitCode=0 Jul 24 13:35:29.001634 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:29.001588 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" Jul 24 13:35:29.001634 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:29.001614 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" event={"ID":"cae77048-0040-4110-924f-100e406cf4f8","Type":"ContainerDied","Data":"9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177"} Jul 24 13:35:29.001924 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:29.001655 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz" event={"ID":"cae77048-0040-4110-924f-100e406cf4f8","Type":"ContainerDied","Data":"e33f6efced305a3eedb07e145c6312cee16322492917b0cd938560c2503770f5"} Jul 24 13:35:29.001924 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:29.001674 2554 scope.go:117] "RemoveContainer" containerID="9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177" Jul 24 13:35:29.010869 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:29.010854 2554 scope.go:117] "RemoveContainer" containerID="9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177" Jul 24 13:35:29.011126 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:35:29.011105 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177\": container with ID starting with 9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177 not found: ID does not exist" containerID="9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177" Jul 24 13:35:29.011171 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:29.011136 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177"} err="failed to get container status \"9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177\": rpc error: code = NotFound desc = could not find container \"9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177\": container with ID starting with 9028f247ae0c0df421705efa644f7c3fa8a0d13a6ccf01562f61623f2394e177 not found: ID does not exist" Jul 24 13:35:29.026709 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:29.026688 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz"] Jul 24 13:35:29.031445 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:29.031423 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b1a3df/auth-test-model-b1a3df-kserve-578897b6c9-zftlz"] Jul 24 13:35:30.598656 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:30.598625 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae77048-0040-4110-924f-100e406cf4f8" path="/var/lib/kubelet/pods/cae77048-0040-4110-924f-100e406cf4f8/volumes" Jul 24 13:35:43.056532 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:43.056499 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:35:46.656300 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:35:46.656264 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:36:00.662996 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:00.662960 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:36:18.687105 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:18.687074 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l"] Jul 24 13:36:18.720149 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:18.720121 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:36:19.925267 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:19.925203 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:36:22.165007 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:22.164961 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:36:25.582118 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:25.582034 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:36:33.706111 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:33.706069 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" podUID="0ca45f63-da56-48b0-839f-9a2e279842c1" containerName="main" containerID="cri-o://04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f" gracePeriod=45 Jul 24 13:36:33.951117 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:33.951092 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:36:34.089901 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.089869 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-home\") pod \"0ca45f63-da56-48b0-839f-9a2e279842c1\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " Jul 24 13:36:34.089901 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.089907 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca45f63-da56-48b0-839f-9a2e279842c1-tls-certs\") pod \"0ca45f63-da56-48b0-839f-9a2e279842c1\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " Jul 24 13:36:34.090167 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.089939 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-dshm\") pod \"0ca45f63-da56-48b0-839f-9a2e279842c1\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " Jul 24 13:36:34.090167 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.089962 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppmr4\" (UniqueName: \"kubernetes.io/projected/0ca45f63-da56-48b0-839f-9a2e279842c1-kube-api-access-ppmr4\") pod \"0ca45f63-da56-48b0-839f-9a2e279842c1\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " Jul 24 13:36:34.090167 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.090003 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-model-cache\") pod \"0ca45f63-da56-48b0-839f-9a2e279842c1\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " Jul 24 13:36:34.090167 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.090054 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-tmp-dir\") pod \"0ca45f63-da56-48b0-839f-9a2e279842c1\" (UID: \"0ca45f63-da56-48b0-839f-9a2e279842c1\") " Jul 24 13:36:34.090365 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.090225 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-home" (OuterVolumeSpecName: "home") pod "0ca45f63-da56-48b0-839f-9a2e279842c1" (UID: "0ca45f63-da56-48b0-839f-9a2e279842c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:36:34.090365 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.090314 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ca45f63-da56-48b0-839f-9a2e279842c1" (UID: "0ca45f63-da56-48b0-839f-9a2e279842c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:36:34.090365 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.090335 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ca45f63-da56-48b0-839f-9a2e279842c1" (UID: "0ca45f63-da56-48b0-839f-9a2e279842c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:36:34.092457 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.092434 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-dshm" (OuterVolumeSpecName: "dshm") pod "0ca45f63-da56-48b0-839f-9a2e279842c1" (UID: "0ca45f63-da56-48b0-839f-9a2e279842c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:36:34.092457 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.092435 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca45f63-da56-48b0-839f-9a2e279842c1-kube-api-access-ppmr4" (OuterVolumeSpecName: "kube-api-access-ppmr4") pod "0ca45f63-da56-48b0-839f-9a2e279842c1" (UID: "0ca45f63-da56-48b0-839f-9a2e279842c1"). InnerVolumeSpecName "kube-api-access-ppmr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:36:34.092651 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.092481 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca45f63-da56-48b0-839f-9a2e279842c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ca45f63-da56-48b0-839f-9a2e279842c1" (UID: "0ca45f63-da56-48b0-839f-9a2e279842c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:36:34.191215 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.191182 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-home\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:36:34.191215 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.191211 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca45f63-da56-48b0-839f-9a2e279842c1-tls-certs\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:36:34.191215 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.191221 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-dshm\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:36:34.191440 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.191229 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppmr4\" (UniqueName: \"kubernetes.io/projected/0ca45f63-da56-48b0-839f-9a2e279842c1-kube-api-access-ppmr4\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:36:34.191440 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.191240 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-model-cache\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:36:34.191440 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.191250 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca45f63-da56-48b0-839f-9a2e279842c1-tmp-dir\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:36:34.278943 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.278907 2554 generic.go:356] "Generic (PLEG): container finished" podID="0ca45f63-da56-48b0-839f-9a2e279842c1" containerID="04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f" exitCode=0 Jul 24 13:36:34.279093 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.278952 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" Jul 24 13:36:34.279093 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.278970 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" event={"ID":"0ca45f63-da56-48b0-839f-9a2e279842c1","Type":"ContainerDied","Data":"04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f"} Jul 24 13:36:34.279093 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.279009 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l" event={"ID":"0ca45f63-da56-48b0-839f-9a2e279842c1","Type":"ContainerDied","Data":"1f5f47da41cd90adc597e6b017e73b8ba9d880d20f28193c5100314253fc8bae"} Jul 24 13:36:34.279093 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.279031 2554 scope.go:117] "RemoveContainer" containerID="04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f" Jul 24 13:36:34.288071 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.288048 2554 scope.go:117] "RemoveContainer" containerID="04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f" Jul 24 13:36:34.288342 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:36:34.288323 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f\": container with ID starting with 04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f not found: ID does not exist" containerID="04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f" Jul 24 13:36:34.288407 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.288354 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f"} err="failed to get container status \"04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f\": rpc error: code = NotFound desc = could not find container \"04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f\": container with ID starting with 04f91c8b3648cdaa993271096bb1c06320be8a4929b480edeb2c27a5f9127b4f not found: ID does not exist" Jul 24 13:36:34.302479 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.302455 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l"] Jul 24 13:36:34.314735 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.314713 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f3a913/sub-test-model-f3a913-kserve-7f5bd4766f-f9g2l"] Jul 24 13:36:34.598025 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:34.597993 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ca45f63-da56-48b0-839f-9a2e279842c1" path="/var/lib/kubelet/pods/0ca45f63-da56-48b0-839f-9a2e279842c1/volumes" Jul 24 13:36:35.259322 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:35.259287 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:36:46.841488 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:46.841454 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:36:50.257170 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:36:50.257133 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:37:08.683080 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:08.683045 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:37:19.973030 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:19.972997 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:23.750259 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:23.750226 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:37:24.855215 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:24.855184 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:24.954927 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:24.954895 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:28.768206 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:28.768173 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:28.887630 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:28.887599 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:31.013468 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:31.013422 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:31.735205 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:31.735166 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:32.674437 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.674408 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7711-139760054609328"] Jul 24 13:37:32.674830 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.674749 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca45f63-da56-48b0-839f-9a2e279842c1" containerName="main" Jul 24 13:37:32.674830 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.674762 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca45f63-da56-48b0-839f-9a2e279842c1" containerName="main" Jul 24 13:37:32.674830 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.674776 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae77048-0040-4110-924f-100e406cf4f8" containerName="main" Jul 24 13:37:32.674830 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.674781 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae77048-0040-4110-924f-100e406cf4f8" containerName="main" Jul 24 13:37:32.674830 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.674830 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae77048-0040-4110-924f-100e406cf4f8" containerName="main" Jul 24 13:37:32.675009 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.674838 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca45f63-da56-48b0-839f-9a2e279842c1" containerName="main" Jul 24 13:37:32.677874 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.677859 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054609328" Jul 24 13:37:32.680616 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.680593 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fkqlt\"" Jul 24 13:37:32.686949 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.686929 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760054609328"] Jul 24 13:37:32.795326 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.795283 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkxrp\" (UniqueName: \"kubernetes.io/projected/74de34aa-4662-48ca-a3cb-61be6cfcc30a-kube-api-access-hkxrp\") pod \"test-curl-7711-139760054609328\" (UID: \"74de34aa-4662-48ca-a3cb-61be6cfcc30a\") " pod="opendatahub/test-curl-7711-139760054609328" Jul 24 13:37:32.896101 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.896073 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkxrp\" (UniqueName: \"kubernetes.io/projected/74de34aa-4662-48ca-a3cb-61be6cfcc30a-kube-api-access-hkxrp\") pod \"test-curl-7711-139760054609328\" (UID: \"74de34aa-4662-48ca-a3cb-61be6cfcc30a\") " pod="opendatahub/test-curl-7711-139760054609328" Jul 24 13:37:32.907457 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.907431 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkxrp\" (UniqueName: \"kubernetes.io/projected/74de34aa-4662-48ca-a3cb-61be6cfcc30a-kube-api-access-hkxrp\") pod \"test-curl-7711-139760054609328\" (UID: \"74de34aa-4662-48ca-a3cb-61be6cfcc30a\") " pod="opendatahub/test-curl-7711-139760054609328" Jul 24 13:37:32.987312 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:32.987233 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054609328" Jul 24 13:37:33.325483 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:33.325379 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760054609328"] Jul 24 13:37:33.328116 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:37:33.328088 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74de34aa_4662_48ca_a3cb_61be6cfcc30a.slice/crio-2e6badb86b711f26f8d74b15dad62ec9a19bdacce3b20770df792ffe0e93097c WatchSource:0}: Error finding container 2e6badb86b711f26f8d74b15dad62ec9a19bdacce3b20770df792ffe0e93097c: Status 404 returned error can't find the container with id 2e6badb86b711f26f8d74b15dad62ec9a19bdacce3b20770df792ffe0e93097c Jul 24 13:37:33.329428 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:33.329411 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 13:37:33.527516 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:33.527479 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760054609328" event={"ID":"74de34aa-4662-48ca-a3cb-61be6cfcc30a","Type":"ContainerStarted","Data":"2e6badb86b711f26f8d74b15dad62ec9a19bdacce3b20770df792ffe0e93097c"} Jul 24 13:37:34.532957 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:34.532923 2554 generic.go:356] "Generic (PLEG): container finished" podID="74de34aa-4662-48ca-a3cb-61be6cfcc30a" containerID="aefee96f530836458faea97d973ca2beaf803eac03d75a2c3d38788f00d48e2c" exitCode=0 Jul 24 13:37:34.533356 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:34.533017 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760054609328" event={"ID":"74de34aa-4662-48ca-a3cb-61be6cfcc30a","Type":"ContainerDied","Data":"aefee96f530836458faea97d973ca2beaf803eac03d75a2c3d38788f00d48e2c"} Jul 24 13:37:35.677200 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.677171 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054609328" Jul 24 13:37:35.710417 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.710393 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7711-139760054609328_74de34aa-4662-48ca-a3cb-61be6cfcc30a/test-curl-7711-139760054609328/0.log" Jul 24 13:37:35.738016 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.737991 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7711-139760054609328"] Jul 24 13:37:35.745648 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.745627 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7711-139760054609328"] Jul 24 13:37:35.823151 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.823064 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkxrp\" (UniqueName: \"kubernetes.io/projected/74de34aa-4662-48ca-a3cb-61be6cfcc30a-kube-api-access-hkxrp\") pod \"74de34aa-4662-48ca-a3cb-61be6cfcc30a\" (UID: \"74de34aa-4662-48ca-a3cb-61be6cfcc30a\") " Jul 24 13:37:35.825547 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.825515 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74de34aa-4662-48ca-a3cb-61be6cfcc30a-kube-api-access-hkxrp" (OuterVolumeSpecName: "kube-api-access-hkxrp") pod "74de34aa-4662-48ca-a3cb-61be6cfcc30a" (UID: "74de34aa-4662-48ca-a3cb-61be6cfcc30a"). InnerVolumeSpecName "kube-api-access-hkxrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:37:35.876431 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.876398 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7711-139760054611120"] Jul 24 13:37:35.876737 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.876724 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74de34aa-4662-48ca-a3cb-61be6cfcc30a" containerName="test-curl-7711-139760054609328" Jul 24 13:37:35.876737 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.876738 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74de34aa-4662-48ca-a3cb-61be6cfcc30a" containerName="test-curl-7711-139760054609328" Jul 24 13:37:35.876824 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.876797 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74de34aa-4662-48ca-a3cb-61be6cfcc30a" containerName="test-curl-7711-139760054609328" Jul 24 13:37:35.880996 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.880980 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054611120" Jul 24 13:37:35.886638 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.886616 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760054611120"] Jul 24 13:37:35.923708 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:35.923674 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkxrp\" (UniqueName: \"kubernetes.io/projected/74de34aa-4662-48ca-a3cb-61be6cfcc30a-kube-api-access-hkxrp\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:37:36.025172 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:36.025138 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldch4\" (UniqueName: \"kubernetes.io/projected/15edff44-d1f3-4ea8-a2e0-6e75004fdc90-kube-api-access-ldch4\") pod \"test-curl-7711-139760054611120\" (UID: \"15edff44-d1f3-4ea8-a2e0-6e75004fdc90\") " pod="opendatahub/test-curl-7711-139760054611120" Jul 24 13:37:36.126639 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:36.126529 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldch4\" (UniqueName: \"kubernetes.io/projected/15edff44-d1f3-4ea8-a2e0-6e75004fdc90-kube-api-access-ldch4\") pod \"test-curl-7711-139760054611120\" (UID: \"15edff44-d1f3-4ea8-a2e0-6e75004fdc90\") " pod="opendatahub/test-curl-7711-139760054611120" Jul 24 13:37:36.139914 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:36.139880 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldch4\" (UniqueName: \"kubernetes.io/projected/15edff44-d1f3-4ea8-a2e0-6e75004fdc90-kube-api-access-ldch4\") pod \"test-curl-7711-139760054611120\" (UID: \"15edff44-d1f3-4ea8-a2e0-6e75004fdc90\") " pod="opendatahub/test-curl-7711-139760054611120" Jul 24 13:37:36.190789 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:36.190754 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054611120" Jul 24 13:37:36.328483 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:36.328386 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760054611120"] Jul 24 13:37:36.330641 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:37:36.330614 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15edff44_d1f3_4ea8_a2e0_6e75004fdc90.slice/crio-8da0f762f2fed1a7f2697e8c5682201e3e0d3cf25d9c872aade6f3a3b672d532 WatchSource:0}: Error finding container 8da0f762f2fed1a7f2697e8c5682201e3e0d3cf25d9c872aade6f3a3b672d532: Status 404 returned error can't find the container with id 8da0f762f2fed1a7f2697e8c5682201e3e0d3cf25d9c872aade6f3a3b672d532 Jul 24 13:37:36.542984 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:36.542956 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054609328" Jul 24 13:37:36.542984 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:36.542980 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e6badb86b711f26f8d74b15dad62ec9a19bdacce3b20770df792ffe0e93097c" Jul 24 13:37:36.544171 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:36.544141 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760054611120" event={"ID":"15edff44-d1f3-4ea8-a2e0-6e75004fdc90","Type":"ContainerStarted","Data":"8da0f762f2fed1a7f2697e8c5682201e3e0d3cf25d9c872aade6f3a3b672d532"} Jul 24 13:37:36.554458 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:37:36.554430 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-139760054609328\" is forbidden: User \"system:node:ip-10-0-134-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object" podUID="74de34aa-4662-48ca-a3cb-61be6cfcc30a" pod="opendatahub/test-curl-7711-139760054609328" Jul 24 13:37:36.598219 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:36.598185 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74de34aa-4662-48ca-a3cb-61be6cfcc30a" path="/var/lib/kubelet/pods/74de34aa-4662-48ca-a3cb-61be6cfcc30a/volumes" Jul 24 13:37:37.550180 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:37.550142 2554 generic.go:356] "Generic (PLEG): container finished" podID="15edff44-d1f3-4ea8-a2e0-6e75004fdc90" containerID="9608aaded3cf01f72debf0e7e4f61b01c7dc9e021015225ec9fb5778349e40ce" exitCode=0 Jul 24 13:37:37.550603 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:37.550218 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760054611120" event={"ID":"15edff44-d1f3-4ea8-a2e0-6e75004fdc90","Type":"ContainerDied","Data":"9608aaded3cf01f72debf0e7e4f61b01c7dc9e021015225ec9fb5778349e40ce"} Jul 24 13:37:38.690303 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.690279 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054611120" Jul 24 13:37:38.715029 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.715000 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7711-139760054611120_15edff44-d1f3-4ea8-a2e0-6e75004fdc90/test-curl-7711-139760054611120/0.log" Jul 24 13:37:38.741729 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.741690 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7711-139760054611120"] Jul 24 13:37:38.748020 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.747988 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldch4\" (UniqueName: \"kubernetes.io/projected/15edff44-d1f3-4ea8-a2e0-6e75004fdc90-kube-api-access-ldch4\") pod \"15edff44-d1f3-4ea8-a2e0-6e75004fdc90\" (UID: \"15edff44-d1f3-4ea8-a2e0-6e75004fdc90\") " Jul 24 13:37:38.748436 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.748412 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7711-139760054611120"] Jul 24 13:37:38.750221 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.750200 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15edff44-d1f3-4ea8-a2e0-6e75004fdc90-kube-api-access-ldch4" (OuterVolumeSpecName: "kube-api-access-ldch4") pod "15edff44-d1f3-4ea8-a2e0-6e75004fdc90" (UID: "15edff44-d1f3-4ea8-a2e0-6e75004fdc90"). InnerVolumeSpecName "kube-api-access-ldch4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:37:38.849117 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.849037 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldch4\" (UniqueName: \"kubernetes.io/projected/15edff44-d1f3-4ea8-a2e0-6e75004fdc90-kube-api-access-ldch4\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:37:38.931949 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.931917 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:38.932231 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.932220 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15edff44-d1f3-4ea8-a2e0-6e75004fdc90" containerName="test-curl-7711-139760054611120" Jul 24 13:37:38.932278 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.932233 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="15edff44-d1f3-4ea8-a2e0-6e75004fdc90" containerName="test-curl-7711-139760054611120" Jul 24 13:37:38.932314 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.932289 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="15edff44-d1f3-4ea8-a2e0-6e75004fdc90" containerName="test-curl-7711-139760054611120" Jul 24 13:37:38.934926 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.934907 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:38.949841 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.949809 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g58zq\" (UniqueName: \"kubernetes.io/projected/93324fb7-20c1-4615-b59f-b4cba9121dda-kube-api-access-g58zq\") pod \"test-curl-7711-139760061008320\" (UID: \"93324fb7-20c1-4615-b59f-b4cba9121dda\") " pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:38.951345 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:38.951322 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:39.050386 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:39.050349 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g58zq\" (UniqueName: \"kubernetes.io/projected/93324fb7-20c1-4615-b59f-b4cba9121dda-kube-api-access-g58zq\") pod \"test-curl-7711-139760061008320\" (UID: \"93324fb7-20c1-4615-b59f-b4cba9121dda\") " pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:39.063950 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:39.063922 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g58zq\" (UniqueName: \"kubernetes.io/projected/93324fb7-20c1-4615-b59f-b4cba9121dda-kube-api-access-g58zq\") pod \"test-curl-7711-139760061008320\" (UID: \"93324fb7-20c1-4615-b59f-b4cba9121dda\") " pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:39.243528 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:39.243437 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:39.379512 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:39.379429 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:39.382221 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:37:39.382193 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93324fb7_20c1_4615_b59f_b4cba9121dda.slice/crio-eeff2148f3a65ee12fad37692019087991acf6722fe872cd5fabe842fec59371 WatchSource:0}: Error finding container eeff2148f3a65ee12fad37692019087991acf6722fe872cd5fabe842fec59371: Status 404 returned error can't find the container with id eeff2148f3a65ee12fad37692019087991acf6722fe872cd5fabe842fec59371 Jul 24 13:37:39.560082 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:39.560054 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054611120" Jul 24 13:37:39.560247 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:39.560064 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8da0f762f2fed1a7f2697e8c5682201e3e0d3cf25d9c872aade6f3a3b672d532" Jul 24 13:37:39.561258 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:39.561233 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760061008320" event={"ID":"93324fb7-20c1-4615-b59f-b4cba9121dda","Type":"ContainerStarted","Data":"eeff2148f3a65ee12fad37692019087991acf6722fe872cd5fabe842fec59371"} Jul 24 13:37:39.570834 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:37:39.570807 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-139760054611120\" is forbidden: User \"system:node:ip-10-0-134-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object" podUID="15edff44-d1f3-4ea8-a2e0-6e75004fdc90" pod="opendatahub/test-curl-7711-139760054611120" Jul 24 13:37:40.567607 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:40.567549 2554 generic.go:356] "Generic (PLEG): container finished" podID="93324fb7-20c1-4615-b59f-b4cba9121dda" containerID="79a406bff52152b2d9e6c996772c2d5dfbcdf8d77ffb34db480ca34bbf827815" exitCode=0 Jul 24 13:37:40.567988 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:40.567620 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760061008320" event={"ID":"93324fb7-20c1-4615-b59f-b4cba9121dda","Type":"ContainerDied","Data":"79a406bff52152b2d9e6c996772c2d5dfbcdf8d77ffb34db480ca34bbf827815"} Jul 24 13:37:40.597732 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:37:40.597694 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-139760054611120\" is forbidden: User \"system:node:ip-10-0-134-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object" podUID="15edff44-d1f3-4ea8-a2e0-6e75004fdc90" pod="opendatahub/test-curl-7711-139760054611120" Jul 24 13:37:40.598333 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:40.598312 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15edff44-d1f3-4ea8-a2e0-6e75004fdc90" path="/var/lib/kubelet/pods/15edff44-d1f3-4ea8-a2e0-6e75004fdc90/volumes" Jul 24 13:37:41.707253 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.707226 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:41.736015 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.735989 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7711-139760061008320_93324fb7-20c1-4615-b59f-b4cba9121dda/test-curl-7711-139760061008320/0.log" Jul 24 13:37:41.768450 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.768414 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:41.776457 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.776431 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:41.777062 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.777045 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g58zq\" (UniqueName: \"kubernetes.io/projected/93324fb7-20c1-4615-b59f-b4cba9121dda-kube-api-access-g58zq\") pod \"93324fb7-20c1-4615-b59f-b4cba9121dda\" (UID: \"93324fb7-20c1-4615-b59f-b4cba9121dda\") " Jul 24 13:37:41.779301 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.779275 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93324fb7-20c1-4615-b59f-b4cba9121dda-kube-api-access-g58zq" (OuterVolumeSpecName: "kube-api-access-g58zq") pod "93324fb7-20c1-4615-b59f-b4cba9121dda" (UID: "93324fb7-20c1-4615-b59f-b4cba9121dda"). InnerVolumeSpecName "kube-api-access-g58zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:37:41.878533 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.878450 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g58zq\" (UniqueName: \"kubernetes.io/projected/93324fb7-20c1-4615-b59f-b4cba9121dda-kube-api-access-g58zq\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:37:41.897975 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.897941 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:41.898306 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.898293 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93324fb7-20c1-4615-b59f-b4cba9121dda" containerName="test-curl-7711-139760061008320" Jul 24 13:37:41.898360 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.898308 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="93324fb7-20c1-4615-b59f-b4cba9121dda" containerName="test-curl-7711-139760061008320" Jul 24 13:37:41.898396 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.898359 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="93324fb7-20c1-4615-b59f-b4cba9121dda" containerName="test-curl-7711-139760061008320" Jul 24 13:37:41.901158 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.901142 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:41.913641 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.913615 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:41.979043 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:41.979008 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb9s\" (UniqueName: \"kubernetes.io/projected/b037be69-4319-4894-91e5-f8ef6205dd17-kube-api-access-8mb9s\") pod \"test-curl-7711-139760061008320\" (UID: \"b037be69-4319-4894-91e5-f8ef6205dd17\") " pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:42.080310 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.080274 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb9s\" (UniqueName: \"kubernetes.io/projected/b037be69-4319-4894-91e5-f8ef6205dd17-kube-api-access-8mb9s\") pod \"test-curl-7711-139760061008320\" (UID: \"b037be69-4319-4894-91e5-f8ef6205dd17\") " pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:42.093213 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.093181 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb9s\" (UniqueName: \"kubernetes.io/projected/b037be69-4319-4894-91e5-f8ef6205dd17-kube-api-access-8mb9s\") pod \"test-curl-7711-139760061008320\" (UID: \"b037be69-4319-4894-91e5-f8ef6205dd17\") " pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:42.210865 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.210776 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:42.359768 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.359736 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:42.362573 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:37:42.362531 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb037be69_4319_4894_91e5_f8ef6205dd17.slice/crio-4eb71810a67f982c4478e6bd9dd31d67a75f5f5c0d1d8660e23ff056452490ee WatchSource:0}: Error finding container 4eb71810a67f982c4478e6bd9dd31d67a75f5f5c0d1d8660e23ff056452490ee: Status 404 returned error can't find the container with id 4eb71810a67f982c4478e6bd9dd31d67a75f5f5c0d1d8660e23ff056452490ee Jul 24 13:37:42.566139 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.566103 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:37:42.578153 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.578124 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:42.578313 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.578127 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeff2148f3a65ee12fad37692019087991acf6722fe872cd5fabe842fec59371" Jul 24 13:37:42.579427 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.579403 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760061008320" event={"ID":"b037be69-4319-4894-91e5-f8ef6205dd17","Type":"ContainerStarted","Data":"4eb71810a67f982c4478e6bd9dd31d67a75f5f5c0d1d8660e23ff056452490ee"} Jul 24 13:37:42.590801 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.590770 2554 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7711-139760061008320" oldPodUID="93324fb7-20c1-4615-b59f-b4cba9121dda" podUID="b037be69-4319-4894-91e5-f8ef6205dd17" Jul 24 13:37:42.598493 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:42.598470 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93324fb7-20c1-4615-b59f-b4cba9121dda" path="/var/lib/kubelet/pods/93324fb7-20c1-4615-b59f-b4cba9121dda/volumes" Jul 24 13:37:43.585111 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:43.585073 2554 generic.go:356] "Generic (PLEG): container finished" podID="b037be69-4319-4894-91e5-f8ef6205dd17" containerID="35942bf1dfc5de44e3290c5d21e95ff621059fc4dd4b3c3e416423d5984d1b32" exitCode=0 Jul 24 13:37:43.585522 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:43.585108 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760061008320" event={"ID":"b037be69-4319-4894-91e5-f8ef6205dd17","Type":"ContainerDied","Data":"35942bf1dfc5de44e3290c5d21e95ff621059fc4dd4b3c3e416423d5984d1b32"} Jul 24 13:37:44.724272 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.724243 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:44.752155 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.752125 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7711-139760061008320_b037be69-4319-4894-91e5-f8ef6205dd17/test-curl-7711-139760061008320/0.log" Jul 24 13:37:44.786510 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.786478 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:44.790848 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.790823 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7711-139760061008320"] Jul 24 13:37:44.802368 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.802347 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mb9s\" (UniqueName: \"kubernetes.io/projected/b037be69-4319-4894-91e5-f8ef6205dd17-kube-api-access-8mb9s\") pod \"b037be69-4319-4894-91e5-f8ef6205dd17\" (UID: \"b037be69-4319-4894-91e5-f8ef6205dd17\") " Jul 24 13:37:44.804625 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.804600 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b037be69-4319-4894-91e5-f8ef6205dd17-kube-api-access-8mb9s" (OuterVolumeSpecName: "kube-api-access-8mb9s") pod "b037be69-4319-4894-91e5-f8ef6205dd17" (UID: "b037be69-4319-4894-91e5-f8ef6205dd17"). InnerVolumeSpecName "kube-api-access-8mb9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:37:44.903335 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.903244 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mb9s\" (UniqueName: \"kubernetes.io/projected/b037be69-4319-4894-91e5-f8ef6205dd17-kube-api-access-8mb9s\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:37:44.923059 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.923024 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7711-139760054543824"] Jul 24 13:37:44.923366 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.923353 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b037be69-4319-4894-91e5-f8ef6205dd17" containerName="test-curl-7711-139760061008320" Jul 24 13:37:44.923406 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.923368 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b037be69-4319-4894-91e5-f8ef6205dd17" containerName="test-curl-7711-139760061008320" Jul 24 13:37:44.923441 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.923426 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b037be69-4319-4894-91e5-f8ef6205dd17" containerName="test-curl-7711-139760061008320" Jul 24 13:37:44.926236 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.926220 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054543824" Jul 24 13:37:44.935441 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:44.935417 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760054543824"] Jul 24 13:37:45.003704 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:45.003672 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ksx6\" (UniqueName: \"kubernetes.io/projected/2eac4c38-0f58-4fa7-b181-dd30e2d2e982-kube-api-access-2ksx6\") pod \"test-curl-7711-139760054543824\" (UID: \"2eac4c38-0f58-4fa7-b181-dd30e2d2e982\") " pod="opendatahub/test-curl-7711-139760054543824" Jul 24 13:37:45.104816 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:45.104780 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ksx6\" (UniqueName: \"kubernetes.io/projected/2eac4c38-0f58-4fa7-b181-dd30e2d2e982-kube-api-access-2ksx6\") pod \"test-curl-7711-139760054543824\" (UID: \"2eac4c38-0f58-4fa7-b181-dd30e2d2e982\") " pod="opendatahub/test-curl-7711-139760054543824" Jul 24 13:37:45.117346 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:45.117312 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ksx6\" (UniqueName: \"kubernetes.io/projected/2eac4c38-0f58-4fa7-b181-dd30e2d2e982-kube-api-access-2ksx6\") pod \"test-curl-7711-139760054543824\" (UID: \"2eac4c38-0f58-4fa7-b181-dd30e2d2e982\") " pod="opendatahub/test-curl-7711-139760054543824" Jul 24 13:37:45.237169 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:45.237078 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054543824" Jul 24 13:37:45.376451 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:45.376419 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-139760054543824"] Jul 24 13:37:45.380517 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:37:45.380490 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eac4c38_0f58_4fa7_b181_dd30e2d2e982.slice/crio-61b47f9ff02ca965f9b4f378e1f467430e3e73d5b838eb60554b64376bd7ef84 WatchSource:0}: Error finding container 61b47f9ff02ca965f9b4f378e1f467430e3e73d5b838eb60554b64376bd7ef84: Status 404 returned error can't find the container with id 61b47f9ff02ca965f9b4f378e1f467430e3e73d5b838eb60554b64376bd7ef84 Jul 24 13:37:45.595735 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:45.595700 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760054543824" event={"ID":"2eac4c38-0f58-4fa7-b181-dd30e2d2e982","Type":"ContainerStarted","Data":"61b47f9ff02ca965f9b4f378e1f467430e3e73d5b838eb60554b64376bd7ef84"} Jul 24 13:37:45.596898 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:45.596880 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eb71810a67f982c4478e6bd9dd31d67a75f5f5c0d1d8660e23ff056452490ee" Jul 24 13:37:45.596898 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:45.596890 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:45.607403 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:37:45.607376 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-139760061008320\" is forbidden: User \"system:node:ip-10-0-134-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-100.ec2.internal' and this object" podUID="b037be69-4319-4894-91e5-f8ef6205dd17" pod="opendatahub/test-curl-7711-139760061008320" Jul 24 13:37:46.599121 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:46.599084 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b037be69-4319-4894-91e5-f8ef6205dd17" path="/var/lib/kubelet/pods/b037be69-4319-4894-91e5-f8ef6205dd17/volumes" Jul 24 13:37:46.602111 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:46.602087 2554 generic.go:356] "Generic (PLEG): container finished" podID="2eac4c38-0f58-4fa7-b181-dd30e2d2e982" containerID="4d66f62080e04e132740066b3b5a9bb638705873fb82deb3feb9645e208c27de" exitCode=0 Jul 24 13:37:46.602218 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:46.602173 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-139760054543824" event={"ID":"2eac4c38-0f58-4fa7-b181-dd30e2d2e982","Type":"ContainerDied","Data":"4d66f62080e04e132740066b3b5a9bb638705873fb82deb3feb9645e208c27de"} Jul 24 13:37:47.743951 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:47.743924 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054543824" Jul 24 13:37:47.787493 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:47.787470 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7711-139760054543824_2eac4c38-0f58-4fa7-b181-dd30e2d2e982/test-curl-7711-139760054543824/0.log" Jul 24 13:37:47.821175 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:47.821143 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7711-139760054543824"] Jul 24 13:37:47.827212 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:47.827188 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7711-139760054543824"] Jul 24 13:37:47.827980 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:47.827962 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ksx6\" (UniqueName: \"kubernetes.io/projected/2eac4c38-0f58-4fa7-b181-dd30e2d2e982-kube-api-access-2ksx6\") pod \"2eac4c38-0f58-4fa7-b181-dd30e2d2e982\" (UID: \"2eac4c38-0f58-4fa7-b181-dd30e2d2e982\") " Jul 24 13:37:47.830131 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:47.830105 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eac4c38-0f58-4fa7-b181-dd30e2d2e982-kube-api-access-2ksx6" (OuterVolumeSpecName: "kube-api-access-2ksx6") pod "2eac4c38-0f58-4fa7-b181-dd30e2d2e982" (UID: "2eac4c38-0f58-4fa7-b181-dd30e2d2e982"). InnerVolumeSpecName "kube-api-access-2ksx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:37:47.929076 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:47.928990 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ksx6\" (UniqueName: \"kubernetes.io/projected/2eac4c38-0f58-4fa7-b181-dd30e2d2e982-kube-api-access-2ksx6\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:37:48.598903 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:48.598870 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eac4c38-0f58-4fa7-b181-dd30e2d2e982" path="/var/lib/kubelet/pods/2eac4c38-0f58-4fa7-b181-dd30e2d2e982/volumes" Jul 24 13:37:48.611971 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:48.611951 2554 scope.go:117] "RemoveContainer" containerID="4d66f62080e04e132740066b3b5a9bb638705873fb82deb3feb9645e208c27de" Jul 24 13:37:48.611971 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:48.611961 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-139760054543824" Jul 24 13:37:51.899886 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:51.899856 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:52.219509 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:52.219424 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:52.660161 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:52.660135 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:55.217553 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:55.217475 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:55.314797 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:55.314766 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:55.436284 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:55.436248 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:55.970503 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:55.970470 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:56.194174 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.194139 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:56.296752 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.296723 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:37:56.372983 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.372950 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc"] Jul 24 13:37:56.373264 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.373252 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eac4c38-0f58-4fa7-b181-dd30e2d2e982" containerName="test-curl-7711-139760054543824" Jul 24 13:37:56.373315 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.373266 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eac4c38-0f58-4fa7-b181-dd30e2d2e982" containerName="test-curl-7711-139760054543824" Jul 24 13:37:56.373352 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.373332 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eac4c38-0f58-4fa7-b181-dd30e2d2e982" containerName="test-curl-7711-139760054543824" Jul 24 13:37:56.377662 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.377637 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:37:56.380367 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.380346 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-49d475\"" Jul 24 13:37:56.380462 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.380417 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-49d475-dockercfg-c9x5z\"" Jul 24 13:37:56.388536 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.388511 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc"] Jul 24 13:37:56.465870 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.465834 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n"] Jul 24 13:37:56.469022 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.469000 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:37:56.490570 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.490538 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n"] Jul 24 13:37:56.503773 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.503746 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t8bv\" (UniqueName: \"kubernetes.io/projected/830a8ef3-595f-41a8-8e9f-338b8fd10bff-kube-api-access-7t8bv\") pod \"payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc\" (UID: \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:37:56.503929 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.503849 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830a8ef3-595f-41a8-8e9f-338b8fd10bff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc\" (UID: \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:37:56.604615 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.604520 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzbtz\" (UniqueName: \"kubernetes.io/projected/7fed51bd-253c-4b87-8fcf-8321299b2017-kube-api-access-pzbtz\") pod \"payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n\" (UID: \"7fed51bd-253c-4b87-8fcf-8321299b2017\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:37:56.604615 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.604579 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fed51bd-253c-4b87-8fcf-8321299b2017-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n\" (UID: \"7fed51bd-253c-4b87-8fcf-8321299b2017\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:37:56.604806 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.604642 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830a8ef3-595f-41a8-8e9f-338b8fd10bff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc\" (UID: \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:37:56.604806 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.604696 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t8bv\" (UniqueName: \"kubernetes.io/projected/830a8ef3-595f-41a8-8e9f-338b8fd10bff-kube-api-access-7t8bv\") pod \"payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc\" (UID: \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:37:56.605341 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.605322 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830a8ef3-595f-41a8-8e9f-338b8fd10bff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc\" (UID: \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:37:56.617256 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.617232 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t8bv\" (UniqueName: \"kubernetes.io/projected/830a8ef3-595f-41a8-8e9f-338b8fd10bff-kube-api-access-7t8bv\") pod \"payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc\" (UID: \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:37:56.688331 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.688300 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:37:56.705703 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.705667 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fed51bd-253c-4b87-8fcf-8321299b2017-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n\" (UID: \"7fed51bd-253c-4b87-8fcf-8321299b2017\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:37:56.706065 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.705930 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzbtz\" (UniqueName: \"kubernetes.io/projected/7fed51bd-253c-4b87-8fcf-8321299b2017-kube-api-access-pzbtz\") pod \"payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n\" (UID: \"7fed51bd-253c-4b87-8fcf-8321299b2017\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:37:56.706546 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.706520 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fed51bd-253c-4b87-8fcf-8321299b2017-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n\" (UID: \"7fed51bd-253c-4b87-8fcf-8321299b2017\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:37:56.717688 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.717664 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzbtz\" (UniqueName: \"kubernetes.io/projected/7fed51bd-253c-4b87-8fcf-8321299b2017-kube-api-access-pzbtz\") pod \"payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n\" (UID: \"7fed51bd-253c-4b87-8fcf-8321299b2017\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:37:56.778908 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.778883 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:37:56.839120 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.839088 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc"] Jul 24 13:37:56.842831 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:37:56.842792 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830a8ef3_595f_41a8_8e9f_338b8fd10bff.slice/crio-98fc3c502f4cd0ad53973cb0d6c652c3d0559ed6fab0404d14c9ae866d2fa5d7 WatchSource:0}: Error finding container 98fc3c502f4cd0ad53973cb0d6c652c3d0559ed6fab0404d14c9ae866d2fa5d7: Status 404 returned error can't find the container with id 98fc3c502f4cd0ad53973cb0d6c652c3d0559ed6fab0404d14c9ae866d2fa5d7 Jul 24 13:37:56.930881 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:56.930791 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n"] Jul 24 13:37:56.933428 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:37:56.933396 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fed51bd_253c_4b87_8fcf_8321299b2017.slice/crio-d61523d3ee5a1ed130c4e73b55d54fc1d31bf9c44e4cdee455780e97ca7d5611 WatchSource:0}: Error finding container d61523d3ee5a1ed130c4e73b55d54fc1d31bf9c44e4cdee455780e97ca7d5611: Status 404 returned error can't find the container with id d61523d3ee5a1ed130c4e73b55d54fc1d31bf9c44e4cdee455780e97ca7d5611 Jul 24 13:37:57.652977 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:57.652947 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" event={"ID":"7fed51bd-253c-4b87-8fcf-8321299b2017","Type":"ContainerStarted","Data":"7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321"} Jul 24 13:37:57.653362 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:57.653002 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" event={"ID":"7fed51bd-253c-4b87-8fcf-8321299b2017","Type":"ContainerStarted","Data":"d61523d3ee5a1ed130c4e73b55d54fc1d31bf9c44e4cdee455780e97ca7d5611"} Jul 24 13:37:57.653362 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:57.653078 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:37:57.654483 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:57.654463 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" event={"ID":"830a8ef3-595f-41a8-8e9f-338b8fd10bff","Type":"ContainerStarted","Data":"3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37"} Jul 24 13:37:57.654650 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:57.654509 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:37:57.654650 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:57.654533 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" event={"ID":"830a8ef3-595f-41a8-8e9f-338b8fd10bff","Type":"ContainerStarted","Data":"98fc3c502f4cd0ad53973cb0d6c652c3d0559ed6fab0404d14c9ae866d2fa5d7"} Jul 24 13:37:57.676191 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:57.676151 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" podStartSLOduration=1.676138599 podStartE2EDuration="1.676138599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:37:57.675042982 +0000 UTC m=+2693.678345095" watchObservedRunningTime="2026-07-24 13:37:57.676138599 +0000 UTC m=+2693.679440703" Jul 24 13:37:57.696154 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:57.696107 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" podStartSLOduration=1.696096804 podStartE2EDuration="1.696096804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:37:57.694448172 +0000 UTC m=+2693.697750276" watchObservedRunningTime="2026-07-24 13:37:57.696096804 +0000 UTC m=+2693.699398908" Jul 24 13:37:59.725992 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:37:59.725961 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:03.263508 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:03.263468 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:38:08.663276 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:08.663240 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:38:08.663679 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:08.663325 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:38:10.635499 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:10.635466 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:11.042254 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:11.042221 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:11.536230 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:11.536208 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:16.958069 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:16.958017 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:17.054031 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:17.053988 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:17.537114 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:17.537081 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:18.067966 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.067914 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:18.280099 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.280070 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:18.398929 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.398835 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:18.469202 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.469161 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr"] Jul 24 13:38:18.472670 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.472648 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:18.475154 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.475130 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-472106\"" Jul 24 13:38:18.475280 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.475155 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-472106-dockercfg-np5sb\"" Jul 24 13:38:18.483000 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.482980 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr"] Jul 24 13:38:18.557764 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.557725 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2"] Jul 24 13:38:18.561223 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.561207 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:18.572350 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.572314 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2"] Jul 24 13:38:18.610200 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.610171 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2\" (UID: \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:18.610388 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.610235 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5s4\" (UniqueName: \"kubernetes.io/projected/e5e601e9-e515-4401-ac4f-20341c90e14e-kube-api-access-6d5s4\") pod \"payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr\" (UID: \"e5e601e9-e515-4401-ac4f-20341c90e14e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:18.610388 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.610288 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xf9\" (UniqueName: \"kubernetes.io/projected/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-kube-api-access-p6xf9\") pod \"payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2\" (UID: \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:18.610388 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.610373 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e601e9-e515-4401-ac4f-20341c90e14e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr\" (UID: \"e5e601e9-e515-4401-ac4f-20341c90e14e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:18.710837 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.710740 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2\" (UID: \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:18.710837 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.710795 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5s4\" (UniqueName: \"kubernetes.io/projected/e5e601e9-e515-4401-ac4f-20341c90e14e-kube-api-access-6d5s4\") pod \"payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr\" (UID: \"e5e601e9-e515-4401-ac4f-20341c90e14e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:18.710837 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.710819 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xf9\" (UniqueName: \"kubernetes.io/projected/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-kube-api-access-p6xf9\") pod \"payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2\" (UID: \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:18.711118 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.710854 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e601e9-e515-4401-ac4f-20341c90e14e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr\" (UID: \"e5e601e9-e515-4401-ac4f-20341c90e14e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:18.711478 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.711459 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2\" (UID: \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:18.711538 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.711524 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e601e9-e515-4401-ac4f-20341c90e14e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr\" (UID: \"e5e601e9-e515-4401-ac4f-20341c90e14e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:18.722218 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.722195 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5s4\" (UniqueName: \"kubernetes.io/projected/e5e601e9-e515-4401-ac4f-20341c90e14e-kube-api-access-6d5s4\") pod \"payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr\" (UID: \"e5e601e9-e515-4401-ac4f-20341c90e14e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:18.722352 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.722282 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xf9\" (UniqueName: \"kubernetes.io/projected/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-kube-api-access-p6xf9\") pod \"payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2\" (UID: \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:18.784523 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.784491 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:18.870978 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.870907 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:18.922161 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:18.922136 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr"] Jul 24 13:38:18.924825 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:38:18.924796 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e601e9_e515_4401_ac4f_20341c90e14e.slice/crio-d8c412ef124abdec7ba205ea1e631228cd602dc4672cda0d0186904661c2b892 WatchSource:0}: Error finding container d8c412ef124abdec7ba205ea1e631228cd602dc4672cda0d0186904661c2b892: Status 404 returned error can't find the container with id d8c412ef124abdec7ba205ea1e631228cd602dc4672cda0d0186904661c2b892 Jul 24 13:38:19.026775 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:19.026743 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2"] Jul 24 13:38:19.029606 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:38:19.029574 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3300c4_d9a6_4325_a8a1_6ae20d07d773.slice/crio-c9661c6bcfd55ef249bc0fcf60afcda087715b5279724ba6c3cf1edc808e3279 WatchSource:0}: Error finding container c9661c6bcfd55ef249bc0fcf60afcda087715b5279724ba6c3cf1edc808e3279: Status 404 returned error can't find the container with id c9661c6bcfd55ef249bc0fcf60afcda087715b5279724ba6c3cf1edc808e3279 Jul 24 13:38:19.754344 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:19.754310 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" event={"ID":"e5e601e9-e515-4401-ac4f-20341c90e14e","Type":"ContainerStarted","Data":"d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f"} Jul 24 13:38:19.754983 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:19.754381 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:19.754983 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:19.754409 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" event={"ID":"e5e601e9-e515-4401-ac4f-20341c90e14e","Type":"ContainerStarted","Data":"d8c412ef124abdec7ba205ea1e631228cd602dc4672cda0d0186904661c2b892"} Jul 24 13:38:19.755835 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:19.755814 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" event={"ID":"9c3300c4-d9a6-4325-a8a1-6ae20d07d773","Type":"ContainerStarted","Data":"817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329"} Jul 24 13:38:19.755918 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:19.755853 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" event={"ID":"9c3300c4-d9a6-4325-a8a1-6ae20d07d773","Type":"ContainerStarted","Data":"c9661c6bcfd55ef249bc0fcf60afcda087715b5279724ba6c3cf1edc808e3279"} Jul 24 13:38:19.755981 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:19.755955 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:19.811579 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:19.811517 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" podStartSLOduration=1.811499114 podStartE2EDuration="1.811499114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:19.778058958 +0000 UTC m=+2715.781361063" watchObservedRunningTime="2026-07-24 13:38:19.811499114 +0000 UTC m=+2715.814801222" Jul 24 13:38:19.812901 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:19.812864 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" podStartSLOduration=1.8128507059999999 podStartE2EDuration="1.812850706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:19.811461098 +0000 UTC m=+2715.814763221" watchObservedRunningTime="2026-07-24 13:38:19.812850706 +0000 UTC m=+2715.816152809" Jul 24 13:38:21.355874 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:21.355840 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:38:21.466889 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:21.466837 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:30.763156 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:30.763123 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:38:30.763766 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:30.763192 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:38:38.745723 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:38.745664 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2_9c3300c4-d9a6-4325-a8a1-6ae20d07d773/payload-processing/0.log" Jul 24 13:38:45.638293 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:45.638258 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:38:48.952061 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:38:48.952020 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:39:04.250540 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:04.250504 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:39:15.287067 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:15.287034 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n_7fed51bd-253c-4b87-8fcf-8321299b2017/payload-processing/0.log" Jul 24 13:39:15.537989 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:15.537907 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2_9c3300c4-d9a6-4325-a8a1-6ae20d07d773/payload-processing/0.log" Jul 24 13:39:15.818358 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:15.818273 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:16.165037 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:16.164950 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:16.547169 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:16.547137 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:16.681856 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:16.681829 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:19.770248 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:19.770211 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:19.885791 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:19.885756 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:21.133672 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.133640 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:21.258391 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.258363 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:21.878781 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.878747 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq"] Jul 24 13:39:21.882066 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.882047 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:21.888619 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.888209 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-45ac1f-dockercfg-55n77\"" Jul 24 13:39:21.888701 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.888664 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-45ac1f\"" Jul 24 13:39:21.897545 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.897519 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq"] Jul 24 13:39:21.946092 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.946066 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk"] Jul 24 13:39:21.949298 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.949281 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:21.967438 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.967418 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk"] Jul 24 13:39:21.982553 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.982532 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d43c41-f470-4a67-9a45-6467bfdffb7d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq\" (UID: \"12d43c41-f470-4a67-9a45-6467bfdffb7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:21.982660 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:21.982608 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdt8m\" (UniqueName: \"kubernetes.io/projected/12d43c41-f470-4a67-9a45-6467bfdffb7d-kube-api-access-hdt8m\") pod \"payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq\" (UID: \"12d43c41-f470-4a67-9a45-6467bfdffb7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:22.077624 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.077593 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:22.083809 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.083785 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5c42a4-d2a9-4a21-b8db-a511688c6418-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk\" (UID: \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:22.083892 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.083860 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d43c41-f470-4a67-9a45-6467bfdffb7d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq\" (UID: \"12d43c41-f470-4a67-9a45-6467bfdffb7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:22.083951 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.083896 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z55jx\" (UniqueName: \"kubernetes.io/projected/5e5c42a4-d2a9-4a21-b8db-a511688c6418-kube-api-access-z55jx\") pod \"payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk\" (UID: \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:22.083951 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.083919 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdt8m\" (UniqueName: \"kubernetes.io/projected/12d43c41-f470-4a67-9a45-6467bfdffb7d-kube-api-access-hdt8m\") pod \"payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq\" (UID: \"12d43c41-f470-4a67-9a45-6467bfdffb7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:22.084535 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.084518 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d43c41-f470-4a67-9a45-6467bfdffb7d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq\" (UID: \"12d43c41-f470-4a67-9a45-6467bfdffb7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:22.095699 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.095674 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdt8m\" (UniqueName: \"kubernetes.io/projected/12d43c41-f470-4a67-9a45-6467bfdffb7d-kube-api-access-hdt8m\") pod \"payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq\" (UID: \"12d43c41-f470-4a67-9a45-6467bfdffb7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:22.164166 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.164097 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:22.184964 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.184931 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z55jx\" (UniqueName: \"kubernetes.io/projected/5e5c42a4-d2a9-4a21-b8db-a511688c6418-kube-api-access-z55jx\") pod \"payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk\" (UID: \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:22.185094 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.184983 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5c42a4-d2a9-4a21-b8db-a511688c6418-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk\" (UID: \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:22.185721 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.185705 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5c42a4-d2a9-4a21-b8db-a511688c6418-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk\" (UID: \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:22.191647 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.191631 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:22.195479 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.195461 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z55jx\" (UniqueName: \"kubernetes.io/projected/5e5c42a4-d2a9-4a21-b8db-a511688c6418-kube-api-access-z55jx\") pod \"payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk\" (UID: \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:22.259546 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.259512 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:22.342914 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.342885 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq"] Jul 24 13:39:22.344719 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:39:22.344694 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d43c41_f470_4a67_9a45_6467bfdffb7d.slice/crio-e4925f72273c54046c21f59ebc55b44fad3fdc45dc7f1f0a81c93c8b5c9dfe8a WatchSource:0}: Error finding container e4925f72273c54046c21f59ebc55b44fad3fdc45dc7f1f0a81c93c8b5c9dfe8a: Status 404 returned error can't find the container with id e4925f72273c54046c21f59ebc55b44fad3fdc45dc7f1f0a81c93c8b5c9dfe8a Jul 24 13:39:22.639150 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:22.639116 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk"] Jul 24 13:39:22.640242 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:39:22.640212 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5c42a4_d2a9_4a21_b8db_a511688c6418.slice/crio-952003fab58cf0d2cc6c5c40da62ee55bc2b7ec86162031eb8fa7850597aa34f WatchSource:0}: Error finding container 952003fab58cf0d2cc6c5c40da62ee55bc2b7ec86162031eb8fa7850597aa34f: Status 404 returned error can't find the container with id 952003fab58cf0d2cc6c5c40da62ee55bc2b7ec86162031eb8fa7850597aa34f Jul 24 13:39:23.061455 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:23.061420 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" event={"ID":"5e5c42a4-d2a9-4a21-b8db-a511688c6418","Type":"ContainerStarted","Data":"10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1"} Jul 24 13:39:23.061672 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:23.061483 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" event={"ID":"5e5c42a4-d2a9-4a21-b8db-a511688c6418","Type":"ContainerStarted","Data":"952003fab58cf0d2cc6c5c40da62ee55bc2b7ec86162031eb8fa7850597aa34f"} Jul 24 13:39:23.061672 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:23.061512 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:23.062967 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:23.062942 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" event={"ID":"12d43c41-f470-4a67-9a45-6467bfdffb7d","Type":"ContainerStarted","Data":"ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63"} Jul 24 13:39:23.063095 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:23.062982 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" event={"ID":"12d43c41-f470-4a67-9a45-6467bfdffb7d","Type":"ContainerStarted","Data":"e4925f72273c54046c21f59ebc55b44fad3fdc45dc7f1f0a81c93c8b5c9dfe8a"} Jul 24 13:39:23.063095 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:23.063023 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:23.084125 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:23.084089 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" podStartSLOduration=2.084075051 podStartE2EDuration="2.084075051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:23.08283823 +0000 UTC m=+2779.086140335" watchObservedRunningTime="2026-07-24 13:39:23.084075051 +0000 UTC m=+2779.087377168" Jul 24 13:39:23.105035 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:23.104982 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" podStartSLOduration=2.104965592 podStartE2EDuration="2.104965592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:23.1044772 +0000 UTC m=+2779.107779307" watchObservedRunningTime="2026-07-24 13:39:23.104965592 +0000 UTC m=+2779.108267697" Jul 24 13:39:24.170967 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:24.170933 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:30.254681 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:30.254589 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:39:34.069537 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:34.069504 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:34.069962 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:34.069602 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:45.373373 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.373338 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.402976 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.402947 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq"] Jul 24 13:39:45.403183 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.403158 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" podUID="12d43c41-f470-4a67-9a45-6467bfdffb7d" containerName="payload-processing" containerID="cri-o://ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63" gracePeriod=30 Jul 24 13:39:45.408783 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.408755 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk"] Jul 24 13:39:45.409045 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.409000 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" podUID="5e5c42a4-d2a9-4a21-b8db-a511688c6418" containerName="payload-pre-processing" containerID="cri-o://10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1" gracePeriod=30 Jul 24 13:39:45.447972 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.447941 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.515045 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.515021 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.687783 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.687765 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:45.690522 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.690506 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:45.718775 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.718754 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z55jx\" (UniqueName: \"kubernetes.io/projected/5e5c42a4-d2a9-4a21-b8db-a511688c6418-kube-api-access-z55jx\") pod \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\" (UID: \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\") " Jul 24 13:39:45.718906 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.718793 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdt8m\" (UniqueName: \"kubernetes.io/projected/12d43c41-f470-4a67-9a45-6467bfdffb7d-kube-api-access-hdt8m\") pod \"12d43c41-f470-4a67-9a45-6467bfdffb7d\" (UID: \"12d43c41-f470-4a67-9a45-6467bfdffb7d\") " Jul 24 13:39:45.718906 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.718831 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5c42a4-d2a9-4a21-b8db-a511688c6418-plugins-config-volume\") pod \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\" (UID: \"5e5c42a4-d2a9-4a21-b8db-a511688c6418\") " Jul 24 13:39:45.718906 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.718902 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d43c41-f470-4a67-9a45-6467bfdffb7d-plugins-config-volume\") pod \"12d43c41-f470-4a67-9a45-6467bfdffb7d\" (UID: \"12d43c41-f470-4a67-9a45-6467bfdffb7d\") " Jul 24 13:39:45.719271 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.719243 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d43c41-f470-4a67-9a45-6467bfdffb7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12d43c41-f470-4a67-9a45-6467bfdffb7d" (UID: "12d43c41-f470-4a67-9a45-6467bfdffb7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:39:45.719271 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.719261 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5c42a4-d2a9-4a21-b8db-a511688c6418-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e5c42a4-d2a9-4a21-b8db-a511688c6418" (UID: "5e5c42a4-d2a9-4a21-b8db-a511688c6418"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:39:45.720982 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.720962 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d43c41-f470-4a67-9a45-6467bfdffb7d-kube-api-access-hdt8m" (OuterVolumeSpecName: "kube-api-access-hdt8m") pod "12d43c41-f470-4a67-9a45-6467bfdffb7d" (UID: "12d43c41-f470-4a67-9a45-6467bfdffb7d"). InnerVolumeSpecName "kube-api-access-hdt8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:39:45.721062 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.721024 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5c42a4-d2a9-4a21-b8db-a511688c6418-kube-api-access-z55jx" (OuterVolumeSpecName: "kube-api-access-z55jx") pod "5e5c42a4-d2a9-4a21-b8db-a511688c6418" (UID: "5e5c42a4-d2a9-4a21-b8db-a511688c6418"). InnerVolumeSpecName "kube-api-access-z55jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:39:45.820316 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.820282 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdt8m\" (UniqueName: \"kubernetes.io/projected/12d43c41-f470-4a67-9a45-6467bfdffb7d-kube-api-access-hdt8m\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:39:45.820316 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.820308 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5c42a4-d2a9-4a21-b8db-a511688c6418-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:39:45.820316 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.820320 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d43c41-f470-4a67-9a45-6467bfdffb7d-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:39:45.820538 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:45.820329 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z55jx\" (UniqueName: \"kubernetes.io/projected/5e5c42a4-d2a9-4a21-b8db-a511688c6418-kube-api-access-z55jx\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:39:46.190965 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.190929 2554 generic.go:356] "Generic (PLEG): container finished" podID="5e5c42a4-d2a9-4a21-b8db-a511688c6418" containerID="10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1" exitCode=0 Jul 24 13:39:46.191176 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.190977 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" Jul 24 13:39:46.191176 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.191019 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" event={"ID":"5e5c42a4-d2a9-4a21-b8db-a511688c6418","Type":"ContainerDied","Data":"10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1"} Jul 24 13:39:46.191176 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.191080 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk" event={"ID":"5e5c42a4-d2a9-4a21-b8db-a511688c6418","Type":"ContainerDied","Data":"952003fab58cf0d2cc6c5c40da62ee55bc2b7ec86162031eb8fa7850597aa34f"} Jul 24 13:39:46.191176 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.191110 2554 scope.go:117] "RemoveContainer" containerID="10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1" Jul 24 13:39:46.192288 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.192234 2554 generic.go:356] "Generic (PLEG): container finished" podID="12d43c41-f470-4a67-9a45-6467bfdffb7d" containerID="ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63" exitCode=0 Jul 24 13:39:46.192365 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.192308 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" event={"ID":"12d43c41-f470-4a67-9a45-6467bfdffb7d","Type":"ContainerDied","Data":"ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63"} Jul 24 13:39:46.192365 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.192325 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" Jul 24 13:39:46.192365 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.192348 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq" event={"ID":"12d43c41-f470-4a67-9a45-6467bfdffb7d","Type":"ContainerDied","Data":"e4925f72273c54046c21f59ebc55b44fad3fdc45dc7f1f0a81c93c8b5c9dfe8a"} Jul 24 13:39:46.201814 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.201792 2554 scope.go:117] "RemoveContainer" containerID="10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1" Jul 24 13:39:46.202181 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:39:46.202056 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1\": container with ID starting with 10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1 not found: ID does not exist" containerID="10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1" Jul 24 13:39:46.202181 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.202084 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1"} err="failed to get container status \"10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1\": rpc error: code = NotFound desc = could not find container \"10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1\": container with ID starting with 10edf2e14aa44f083a13562c59641fa4d49c5a37844d601b1a2f1802f2e524a1 not found: ID does not exist" Jul 24 13:39:46.202181 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.202100 2554 scope.go:117] "RemoveContainer" containerID="ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63" Jul 24 13:39:46.210136 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.210120 2554 scope.go:117] "RemoveContainer" containerID="ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63" Jul 24 13:39:46.210368 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:39:46.210350 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63\": container with ID starting with ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63 not found: ID does not exist" containerID="ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63" Jul 24 13:39:46.210419 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.210373 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63"} err="failed to get container status \"ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63\": rpc error: code = NotFound desc = could not find container \"ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63\": container with ID starting with ff361da71e4f987f72ad04b865f2ca20be1662bcbe8a4e5ef80c020227b19d63 not found: ID does not exist" Jul 24 13:39:46.217730 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.217703 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk"] Jul 24 13:39:46.222860 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.222839 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-45ac1f-6f647f4f84-pvzlk"] Jul 24 13:39:46.234942 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.234920 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq"] Jul 24 13:39:46.240855 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.240837 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-45ac1f-57676d6fbd-2sksq"] Jul 24 13:39:46.599765 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.599733 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12d43c41-f470-4a67-9a45-6467bfdffb7d" path="/var/lib/kubelet/pods/12d43c41-f470-4a67-9a45-6467bfdffb7d/volumes" Jul 24 13:39:46.600151 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:46.600089 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e5c42a4-d2a9-4a21-b8db-a511688c6418" path="/var/lib/kubelet/pods/5e5c42a4-d2a9-4a21-b8db-a511688c6418/volumes" Jul 24 13:39:50.195037 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:50.195005 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:50.597766 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:50.597723 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:52.967385 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:52.967351 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:39:57.305039 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:57.305010 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:57.412295 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:57.412265 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:57.600407 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:57.600326 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:57.830115 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:39:57.830075 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:05.977261 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:05.977228 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:06.070992 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:06.070953 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:16.462531 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:16.462491 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:40:18.383339 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.383308 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:18.427989 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.427957 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n"] Jul 24 13:40:18.428206 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.428183 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" podUID="7fed51bd-253c-4b87-8fcf-8321299b2017" containerName="payload-processing" containerID="cri-o://7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321" gracePeriod=30 Jul 24 13:40:18.437975 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.437693 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc"] Jul 24 13:40:18.437975 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.437935 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" podUID="830a8ef3-595f-41a8-8e9f-338b8fd10bff" containerName="payload-pre-processing" containerID="cri-o://3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37" gracePeriod=30 Jul 24 13:40:18.475088 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.475049 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:18.576667 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.576627 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:18.692414 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.692392 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:40:18.727799 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.727777 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:40:18.813921 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.813892 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t8bv\" (UniqueName: \"kubernetes.io/projected/830a8ef3-595f-41a8-8e9f-338b8fd10bff-kube-api-access-7t8bv\") pod \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\" (UID: \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\") " Jul 24 13:40:18.813921 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.813929 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzbtz\" (UniqueName: \"kubernetes.io/projected/7fed51bd-253c-4b87-8fcf-8321299b2017-kube-api-access-pzbtz\") pod \"7fed51bd-253c-4b87-8fcf-8321299b2017\" (UID: \"7fed51bd-253c-4b87-8fcf-8321299b2017\") " Jul 24 13:40:18.814138 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.813955 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830a8ef3-595f-41a8-8e9f-338b8fd10bff-plugins-config-volume\") pod \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\" (UID: \"830a8ef3-595f-41a8-8e9f-338b8fd10bff\") " Jul 24 13:40:18.814138 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.813972 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fed51bd-253c-4b87-8fcf-8321299b2017-plugins-config-volume\") pod \"7fed51bd-253c-4b87-8fcf-8321299b2017\" (UID: \"7fed51bd-253c-4b87-8fcf-8321299b2017\") " Jul 24 13:40:18.814345 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.814326 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830a8ef3-595f-41a8-8e9f-338b8fd10bff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "830a8ef3-595f-41a8-8e9f-338b8fd10bff" (UID: "830a8ef3-595f-41a8-8e9f-338b8fd10bff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:40:18.814391 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.814354 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fed51bd-253c-4b87-8fcf-8321299b2017-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7fed51bd-253c-4b87-8fcf-8321299b2017" (UID: "7fed51bd-253c-4b87-8fcf-8321299b2017"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:40:18.816162 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.816137 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fed51bd-253c-4b87-8fcf-8321299b2017-kube-api-access-pzbtz" (OuterVolumeSpecName: "kube-api-access-pzbtz") pod "7fed51bd-253c-4b87-8fcf-8321299b2017" (UID: "7fed51bd-253c-4b87-8fcf-8321299b2017"). InnerVolumeSpecName "kube-api-access-pzbtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:40:18.816265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.816169 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830a8ef3-595f-41a8-8e9f-338b8fd10bff-kube-api-access-7t8bv" (OuterVolumeSpecName: "kube-api-access-7t8bv") pod "830a8ef3-595f-41a8-8e9f-338b8fd10bff" (UID: "830a8ef3-595f-41a8-8e9f-338b8fd10bff"). InnerVolumeSpecName "kube-api-access-7t8bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:40:18.915597 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.915491 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t8bv\" (UniqueName: \"kubernetes.io/projected/830a8ef3-595f-41a8-8e9f-338b8fd10bff-kube-api-access-7t8bv\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:40:18.915597 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.915519 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzbtz\" (UniqueName: \"kubernetes.io/projected/7fed51bd-253c-4b87-8fcf-8321299b2017-kube-api-access-pzbtz\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:40:18.915597 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.915528 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830a8ef3-595f-41a8-8e9f-338b8fd10bff-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:40:18.915597 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:18.915536 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fed51bd-253c-4b87-8fcf-8321299b2017-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:40:19.357331 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.357294 2554 generic.go:356] "Generic (PLEG): container finished" podID="7fed51bd-253c-4b87-8fcf-8321299b2017" containerID="7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321" exitCode=0 Jul 24 13:40:19.357526 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.357345 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" Jul 24 13:40:19.357526 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.357383 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" event={"ID":"7fed51bd-253c-4b87-8fcf-8321299b2017","Type":"ContainerDied","Data":"7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321"} Jul 24 13:40:19.357526 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.357448 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" event={"ID":"7fed51bd-253c-4b87-8fcf-8321299b2017","Type":"ContainerDied","Data":"d61523d3ee5a1ed130c4e73b55d54fc1d31bf9c44e4cdee455780e97ca7d5611"} Jul 24 13:40:19.357526 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.357474 2554 scope.go:117] "RemoveContainer" containerID="7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321" Jul 24 13:40:19.358724 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.358702 2554 generic.go:356] "Generic (PLEG): container finished" podID="830a8ef3-595f-41a8-8e9f-338b8fd10bff" containerID="3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37" exitCode=0 Jul 24 13:40:19.358816 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.358740 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" Jul 24 13:40:19.358816 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.358751 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" event={"ID":"830a8ef3-595f-41a8-8e9f-338b8fd10bff","Type":"ContainerDied","Data":"3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37"} Jul 24 13:40:19.358816 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.358793 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" event={"ID":"830a8ef3-595f-41a8-8e9f-338b8fd10bff","Type":"ContainerDied","Data":"98fc3c502f4cd0ad53973cb0d6c652c3d0559ed6fab0404d14c9ae866d2fa5d7"} Jul 24 13:40:19.366283 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.366167 2554 scope.go:117] "RemoveContainer" containerID="7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321" Jul 24 13:40:19.366649 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:40:19.366624 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321\": container with ID starting with 7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321 not found: ID does not exist" containerID="7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321" Jul 24 13:40:19.366728 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.366673 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321"} err="failed to get container status \"7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321\": rpc error: code = NotFound desc = could not find container \"7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321\": container with ID starting with 7060312b2840e43c7acba8ab92e42f57295ee3209167f7b04275207411f36321 not found: ID does not exist" Jul 24 13:40:19.366728 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.366694 2554 scope.go:117] "RemoveContainer" containerID="3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37" Jul 24 13:40:19.375206 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.375188 2554 scope.go:117] "RemoveContainer" containerID="3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37" Jul 24 13:40:19.375482 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:40:19.375463 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37\": container with ID starting with 3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37 not found: ID does not exist" containerID="3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37" Jul 24 13:40:19.375527 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.375490 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37"} err="failed to get container status \"3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37\": rpc error: code = NotFound desc = could not find container \"3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37\": container with ID starting with 3b591d13855e5cf83cace48f59a9d59eae4742b9e3e45bd43308d44d0f03cd37 not found: ID does not exist" Jul 24 13:40:19.383233 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.383210 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n"] Jul 24 13:40:19.387577 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.387537 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n"] Jul 24 13:40:19.402173 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.402149 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc"] Jul 24 13:40:19.404896 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.404874 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc"] Jul 24 13:40:19.662833 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.662735 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-49d475-74c768b9fb-hgj2n" podUID="7fed51bd-253c-4b87-8fcf-8321299b2017" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.55:9004: i/o timeout" Jul 24 13:40:19.662992 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:19.662750 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-49d475-65d8999d76-2kqtc" podUID="830a8ef3-595f-41a8-8e9f-338b8fd10bff" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.54:9004: i/o timeout" Jul 24 13:40:20.598626 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:20.598590 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fed51bd-253c-4b87-8fcf-8321299b2017" path="/var/lib/kubelet/pods/7fed51bd-253c-4b87-8fcf-8321299b2017/volumes" Jul 24 13:40:20.599004 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:20.598951 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="830a8ef3-595f-41a8-8e9f-338b8fd10bff" path="/var/lib/kubelet/pods/830a8ef3-595f-41a8-8e9f-338b8fd10bff/volumes" Jul 24 13:40:21.558185 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:21.558150 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:21.898878 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:21.898793 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:26.906509 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:26.906472 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:30.067143 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:30.067107 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:40:47.667479 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:40:47.667443 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:04.543779 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:04.543705 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:05.172005 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:05.171909 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:11.653578 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:11.653525 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:11.750311 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:11.750281 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:11.939161 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:11.939079 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:12.177937 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:12.177907 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:22.556286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:22.556249 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:22.917517 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:22.917432 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2"] Jul 24 13:41:22.917726 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:22.917667 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" podUID="9c3300c4-d9a6-4325-a8a1-6ae20d07d773" containerName="payload-processing" containerID="cri-o://817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329" gracePeriod=30 Jul 24 13:41:22.927944 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:22.927914 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr"] Jul 24 13:41:22.928127 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:22.928108 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" podUID="e5e601e9-e515-4401-ac4f-20341c90e14e" containerName="payload-pre-processing" containerID="cri-o://d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f" gracePeriod=30 Jul 24 13:41:23.212954 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.212928 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:41:23.215894 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.215875 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:41:23.267105 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.267076 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6xf9\" (UniqueName: \"kubernetes.io/projected/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-kube-api-access-p6xf9\") pod \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\" (UID: \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\") " Jul 24 13:41:23.267289 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.267139 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-plugins-config-volume\") pod \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\" (UID: \"9c3300c4-d9a6-4325-a8a1-6ae20d07d773\") " Jul 24 13:41:23.267289 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.267160 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e601e9-e515-4401-ac4f-20341c90e14e-plugins-config-volume\") pod \"e5e601e9-e515-4401-ac4f-20341c90e14e\" (UID: \"e5e601e9-e515-4401-ac4f-20341c90e14e\") " Jul 24 13:41:23.267289 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.267192 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d5s4\" (UniqueName: \"kubernetes.io/projected/e5e601e9-e515-4401-ac4f-20341c90e14e-kube-api-access-6d5s4\") pod \"e5e601e9-e515-4401-ac4f-20341c90e14e\" (UID: \"e5e601e9-e515-4401-ac4f-20341c90e14e\") " Jul 24 13:41:23.267604 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.267551 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e601e9-e515-4401-ac4f-20341c90e14e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5e601e9-e515-4401-ac4f-20341c90e14e" (UID: "e5e601e9-e515-4401-ac4f-20341c90e14e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:23.267604 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.267584 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c3300c4-d9a6-4325-a8a1-6ae20d07d773" (UID: "9c3300c4-d9a6-4325-a8a1-6ae20d07d773"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:23.269479 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.269449 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e601e9-e515-4401-ac4f-20341c90e14e-kube-api-access-6d5s4" (OuterVolumeSpecName: "kube-api-access-6d5s4") pod "e5e601e9-e515-4401-ac4f-20341c90e14e" (UID: "e5e601e9-e515-4401-ac4f-20341c90e14e"). InnerVolumeSpecName "kube-api-access-6d5s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:41:23.269603 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.269512 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-kube-api-access-p6xf9" (OuterVolumeSpecName: "kube-api-access-p6xf9") pod "9c3300c4-d9a6-4325-a8a1-6ae20d07d773" (UID: "9c3300c4-d9a6-4325-a8a1-6ae20d07d773"). InnerVolumeSpecName "kube-api-access-p6xf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:41:23.368269 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.368232 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e601e9-e515-4401-ac4f-20341c90e14e-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:41:23.368269 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.368262 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d5s4\" (UniqueName: \"kubernetes.io/projected/e5e601e9-e515-4401-ac4f-20341c90e14e-kube-api-access-6d5s4\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:41:23.368269 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.368274 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6xf9\" (UniqueName: \"kubernetes.io/projected/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-kube-api-access-p6xf9\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:41:23.368505 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.368282 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3300c4-d9a6-4325-a8a1-6ae20d07d773-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:41:23.649195 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.649163 2554 generic.go:356] "Generic (PLEG): container finished" podID="e5e601e9-e515-4401-ac4f-20341c90e14e" containerID="d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f" exitCode=0 Jul 24 13:41:23.649669 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.649216 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" Jul 24 13:41:23.649669 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.649255 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" event={"ID":"e5e601e9-e515-4401-ac4f-20341c90e14e","Type":"ContainerDied","Data":"d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f"} Jul 24 13:41:23.649669 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.649306 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr" event={"ID":"e5e601e9-e515-4401-ac4f-20341c90e14e","Type":"ContainerDied","Data":"d8c412ef124abdec7ba205ea1e631228cd602dc4672cda0d0186904661c2b892"} Jul 24 13:41:23.649669 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.649333 2554 scope.go:117] "RemoveContainer" containerID="d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f" Jul 24 13:41:23.650612 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.650585 2554 generic.go:356] "Generic (PLEG): container finished" podID="9c3300c4-d9a6-4325-a8a1-6ae20d07d773" containerID="817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329" exitCode=0 Jul 24 13:41:23.650717 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.650628 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" Jul 24 13:41:23.650717 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.650658 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" event={"ID":"9c3300c4-d9a6-4325-a8a1-6ae20d07d773","Type":"ContainerDied","Data":"817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329"} Jul 24 13:41:23.650717 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.650691 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2" event={"ID":"9c3300c4-d9a6-4325-a8a1-6ae20d07d773","Type":"ContainerDied","Data":"c9661c6bcfd55ef249bc0fcf60afcda087715b5279724ba6c3cf1edc808e3279"} Jul 24 13:41:23.658806 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.658794 2554 scope.go:117] "RemoveContainer" containerID="d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f" Jul 24 13:41:23.659080 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:41:23.659057 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f\": container with ID starting with d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f not found: ID does not exist" containerID="d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f" Jul 24 13:41:23.659147 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.659093 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f"} err="failed to get container status \"d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f\": rpc error: code = NotFound desc = could not find container \"d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f\": container with ID starting with d6d3776aad2ff57405904f4eace57fa55301c441c11cdbaf6b09542f254c491f not found: ID does not exist" Jul 24 13:41:23.659147 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.659117 2554 scope.go:117] "RemoveContainer" containerID="817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329" Jul 24 13:41:23.666631 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.666615 2554 scope.go:117] "RemoveContainer" containerID="817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329" Jul 24 13:41:23.666896 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:41:23.666879 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329\": container with ID starting with 817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329 not found: ID does not exist" containerID="817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329" Jul 24 13:41:23.666948 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.666903 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329"} err="failed to get container status \"817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329\": rpc error: code = NotFound desc = could not find container \"817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329\": container with ID starting with 817eb1a22a6dd2b2075c7a81a3ccacef1a4e586b1d61efee3ed7d3f3927c4329 not found: ID does not exist" Jul 24 13:41:23.672529 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.672508 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr"] Jul 24 13:41:23.677341 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.677321 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-472106-55747b586d-wthzr"] Jul 24 13:41:23.693454 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.693432 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2"] Jul 24 13:41:23.695395 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.695375 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-472106-7d7977d6d4-h29b2"] Jul 24 13:41:23.766381 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:23.766353 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:24.599008 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:24.598976 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c3300c4-d9a6-4325-a8a1-6ae20d07d773" path="/var/lib/kubelet/pods/9c3300c4-d9a6-4325-a8a1-6ae20d07d773/volumes" Jul 24 13:41:24.599327 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:24.599313 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e601e9-e515-4401-ac4f-20341c90e14e" path="/var/lib/kubelet/pods/e5e601e9-e515-4401-ac4f-20341c90e14e/volumes" Jul 24 13:41:26.263598 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:26.263549 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:41:27.403988 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:27.403953 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:27.787861 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:27.787836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:34.616086 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:34.616055 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:34.716733 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:34.716702 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:34.902006 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:34.901924 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:35.200991 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:35.200906 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:45.884909 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:45.884865 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:45.917956 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:45.917922 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2"] Jul 24 13:41:45.918394 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:45.918346 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" podUID="02063e5e-9e36-4141-9b29-a4c334b8f007" containerName="payload-processing" containerID="cri-o://e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1" gracePeriod=30 Jul 24 13:41:45.923778 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:45.923721 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5"] Jul 24 13:41:45.924000 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:45.923957 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" podUID="8f053ba3-c11d-446c-a33f-70f714678f42" containerName="payload-pre-processing" containerID="cri-o://3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9" gracePeriod=30 Jul 24 13:41:46.014022 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.013985 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:46.114626 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.114596 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:46.193038 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.193013 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:41:46.195911 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.195894 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:41:46.267389 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.267357 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7qqw\" (UniqueName: \"kubernetes.io/projected/02063e5e-9e36-4141-9b29-a4c334b8f007-kube-api-access-m7qqw\") pod \"02063e5e-9e36-4141-9b29-a4c334b8f007\" (UID: \"02063e5e-9e36-4141-9b29-a4c334b8f007\") " Jul 24 13:41:46.267600 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.267397 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f053ba3-c11d-446c-a33f-70f714678f42-plugins-config-volume\") pod \"8f053ba3-c11d-446c-a33f-70f714678f42\" (UID: \"8f053ba3-c11d-446c-a33f-70f714678f42\") " Jul 24 13:41:46.267600 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.267454 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8rr6\" (UniqueName: \"kubernetes.io/projected/8f053ba3-c11d-446c-a33f-70f714678f42-kube-api-access-t8rr6\") pod \"8f053ba3-c11d-446c-a33f-70f714678f42\" (UID: \"8f053ba3-c11d-446c-a33f-70f714678f42\") " Jul 24 13:41:46.267600 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.267487 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02063e5e-9e36-4141-9b29-a4c334b8f007-plugins-config-volume\") pod \"02063e5e-9e36-4141-9b29-a4c334b8f007\" (UID: \"02063e5e-9e36-4141-9b29-a4c334b8f007\") " Jul 24 13:41:46.267914 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.267886 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f053ba3-c11d-446c-a33f-70f714678f42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f053ba3-c11d-446c-a33f-70f714678f42" (UID: "8f053ba3-c11d-446c-a33f-70f714678f42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:46.267975 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.267887 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02063e5e-9e36-4141-9b29-a4c334b8f007-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02063e5e-9e36-4141-9b29-a4c334b8f007" (UID: "02063e5e-9e36-4141-9b29-a4c334b8f007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:46.269762 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.269738 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02063e5e-9e36-4141-9b29-a4c334b8f007-kube-api-access-m7qqw" (OuterVolumeSpecName: "kube-api-access-m7qqw") pod "02063e5e-9e36-4141-9b29-a4c334b8f007" (UID: "02063e5e-9e36-4141-9b29-a4c334b8f007"). InnerVolumeSpecName "kube-api-access-m7qqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:41:46.269869 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.269838 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f053ba3-c11d-446c-a33f-70f714678f42-kube-api-access-t8rr6" (OuterVolumeSpecName: "kube-api-access-t8rr6") pod "8f053ba3-c11d-446c-a33f-70f714678f42" (UID: "8f053ba3-c11d-446c-a33f-70f714678f42"). InnerVolumeSpecName "kube-api-access-t8rr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:41:46.368861 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.368827 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8rr6\" (UniqueName: \"kubernetes.io/projected/8f053ba3-c11d-446c-a33f-70f714678f42-kube-api-access-t8rr6\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:41:46.368861 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.368857 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02063e5e-9e36-4141-9b29-a4c334b8f007-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:41:46.368861 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.368867 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7qqw\" (UniqueName: \"kubernetes.io/projected/02063e5e-9e36-4141-9b29-a4c334b8f007-kube-api-access-m7qqw\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:41:46.369086 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.368875 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f053ba3-c11d-446c-a33f-70f714678f42-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:41:46.754463 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.754426 2554 generic.go:356] "Generic (PLEG): container finished" podID="02063e5e-9e36-4141-9b29-a4c334b8f007" containerID="e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1" exitCode=0 Jul 24 13:41:46.754675 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.754476 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" Jul 24 13:41:46.754675 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.754513 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" event={"ID":"02063e5e-9e36-4141-9b29-a4c334b8f007","Type":"ContainerDied","Data":"e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1"} Jul 24 13:41:46.754675 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.754603 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2" event={"ID":"02063e5e-9e36-4141-9b29-a4c334b8f007","Type":"ContainerDied","Data":"b35699468fbaebe123dcf840ab029d0566270aaeaaaef61596b9a024e1cd19c2"} Jul 24 13:41:46.754675 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.754638 2554 scope.go:117] "RemoveContainer" containerID="e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1" Jul 24 13:41:46.755886 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.755864 2554 generic.go:356] "Generic (PLEG): container finished" podID="8f053ba3-c11d-446c-a33f-70f714678f42" containerID="3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9" exitCode=0 Jul 24 13:41:46.756000 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.755907 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" Jul 24 13:41:46.756000 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.755955 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" event={"ID":"8f053ba3-c11d-446c-a33f-70f714678f42","Type":"ContainerDied","Data":"3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9"} Jul 24 13:41:46.756080 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.756009 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5" event={"ID":"8f053ba3-c11d-446c-a33f-70f714678f42","Type":"ContainerDied","Data":"f1ec072cf7df42b2f398c1c2a7f1d127e6deecaaf6b90f57215e50f14f673e56"} Jul 24 13:41:46.763519 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.763495 2554 scope.go:117] "RemoveContainer" containerID="e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1" Jul 24 13:41:46.763841 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:41:46.763822 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1\": container with ID starting with e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1 not found: ID does not exist" containerID="e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1" Jul 24 13:41:46.763909 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.763850 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1"} err="failed to get container status \"e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1\": rpc error: code = NotFound desc = could not find container \"e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1\": container with ID starting with e164e83a224fd7e5ae1a52e6213ec756e5a85d11cefa3f28fbbbed7df647f3d1 not found: ID does not exist" Jul 24 13:41:46.763909 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.763865 2554 scope.go:117] "RemoveContainer" containerID="3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9" Jul 24 13:41:46.771294 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.771276 2554 scope.go:117] "RemoveContainer" containerID="3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9" Jul 24 13:41:46.771591 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:41:46.771549 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9\": container with ID starting with 3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9 not found: ID does not exist" containerID="3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9" Jul 24 13:41:46.771687 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.771597 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9"} err="failed to get container status \"3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9\": rpc error: code = NotFound desc = could not find container \"3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9\": container with ID starting with 3d51d964fdc8432fa9cf5dce5d5150b743d2d6b0ecc0a4e456603c674141d0f9 not found: ID does not exist" Jul 24 13:41:46.774181 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.774159 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5"] Jul 24 13:41:46.778696 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.778674 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f3a913-84c6d959dd-pw8t5"] Jul 24 13:41:46.792018 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.791996 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2"] Jul 24 13:41:46.796300 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:46.796280 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f3a913-7c8977b854-hflg2"] Jul 24 13:41:48.598688 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:48.598648 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02063e5e-9e36-4141-9b29-a4c334b8f007" path="/var/lib/kubelet/pods/02063e5e-9e36-4141-9b29-a4c334b8f007/volumes" Jul 24 13:41:48.599188 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:48.599166 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f053ba3-c11d-446c-a33f-70f714678f42" path="/var/lib/kubelet/pods/8f053ba3-c11d-446c-a33f-70f714678f42/volumes" Jul 24 13:41:48.950446 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:48.950366 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:41:50.364062 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:50.364032 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:50.780629 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:50.780483 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:57.553948 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:57.553910 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:57.689727 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:57.689687 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:57.825677 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:57.825519 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:58.031856 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:41:58.031761 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:08.824022 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:08.823969 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:08.869411 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:08.869371 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt"] Jul 24 13:42:08.869607 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:08.869583 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" podUID="d862a456-95ed-4f15-a6dc-73bee61f4f3f" containerName="payload-processing" containerID="cri-o://655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a" gracePeriod=30 Jul 24 13:42:08.878904 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:08.878856 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z"] Jul 24 13:42:08.879200 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:08.879118 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" podUID="1d493ddf-88f1-4f51-9be5-da2d64442fcc" containerName="payload-pre-processing" containerID="cri-o://991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc" gracePeriod=30 Jul 24 13:42:08.934895 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:08.934859 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:09.034725 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.034679 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:09.126143 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.126120 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:42:09.158338 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.158314 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:42:09.268811 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.268779 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d862a456-95ed-4f15-a6dc-73bee61f4f3f-plugins-config-volume\") pod \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\" (UID: \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\") " Jul 24 13:42:09.268811 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.268818 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6mll\" (UniqueName: \"kubernetes.io/projected/1d493ddf-88f1-4f51-9be5-da2d64442fcc-kube-api-access-g6mll\") pod \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\" (UID: \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\") " Jul 24 13:42:09.269039 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.268865 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lrcl\" (UniqueName: \"kubernetes.io/projected/d862a456-95ed-4f15-a6dc-73bee61f4f3f-kube-api-access-8lrcl\") pod \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\" (UID: \"d862a456-95ed-4f15-a6dc-73bee61f4f3f\") " Jul 24 13:42:09.269039 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.268907 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d493ddf-88f1-4f51-9be5-da2d64442fcc-plugins-config-volume\") pod \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\" (UID: \"1d493ddf-88f1-4f51-9be5-da2d64442fcc\") " Jul 24 13:42:09.269346 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.269317 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d862a456-95ed-4f15-a6dc-73bee61f4f3f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d862a456-95ed-4f15-a6dc-73bee61f4f3f" (UID: "d862a456-95ed-4f15-a6dc-73bee61f4f3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:42:09.269346 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.269337 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d493ddf-88f1-4f51-9be5-da2d64442fcc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d493ddf-88f1-4f51-9be5-da2d64442fcc" (UID: "1d493ddf-88f1-4f51-9be5-da2d64442fcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:42:09.271116 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.271094 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d862a456-95ed-4f15-a6dc-73bee61f4f3f-kube-api-access-8lrcl" (OuterVolumeSpecName: "kube-api-access-8lrcl") pod "d862a456-95ed-4f15-a6dc-73bee61f4f3f" (UID: "d862a456-95ed-4f15-a6dc-73bee61f4f3f"). InnerVolumeSpecName "kube-api-access-8lrcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:42:09.271116 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.271104 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d493ddf-88f1-4f51-9be5-da2d64442fcc-kube-api-access-g6mll" (OuterVolumeSpecName: "kube-api-access-g6mll") pod "1d493ddf-88f1-4f51-9be5-da2d64442fcc" (UID: "1d493ddf-88f1-4f51-9be5-da2d64442fcc"). InnerVolumeSpecName "kube-api-access-g6mll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:42:09.369831 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.369749 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d493ddf-88f1-4f51-9be5-da2d64442fcc-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:42:09.369831 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.369778 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d862a456-95ed-4f15-a6dc-73bee61f4f3f-plugins-config-volume\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:42:09.369831 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.369787 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mll\" (UniqueName: \"kubernetes.io/projected/1d493ddf-88f1-4f51-9be5-da2d64442fcc-kube-api-access-g6mll\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:42:09.369831 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.369796 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lrcl\" (UniqueName: \"kubernetes.io/projected/d862a456-95ed-4f15-a6dc-73bee61f4f3f-kube-api-access-8lrcl\") on node \"ip-10-0-134-100.ec2.internal\" DevicePath \"\"" Jul 24 13:42:09.858138 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.858102 2554 generic.go:356] "Generic (PLEG): container finished" podID="d862a456-95ed-4f15-a6dc-73bee61f4f3f" containerID="655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a" exitCode=0 Jul 24 13:42:09.858596 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.858153 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" Jul 24 13:42:09.858596 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.858196 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" event={"ID":"d862a456-95ed-4f15-a6dc-73bee61f4f3f","Type":"ContainerDied","Data":"655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a"} Jul 24 13:42:09.858596 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.858258 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt" event={"ID":"d862a456-95ed-4f15-a6dc-73bee61f4f3f","Type":"ContainerDied","Data":"73ed45bb27e280fb896492cefe50185d5182a7651bee0383fdac00967dcf457a"} Jul 24 13:42:09.858596 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.858293 2554 scope.go:117] "RemoveContainer" containerID="655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a" Jul 24 13:42:09.859462 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.859441 2554 generic.go:356] "Generic (PLEG): container finished" podID="1d493ddf-88f1-4f51-9be5-da2d64442fcc" containerID="991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc" exitCode=0 Jul 24 13:42:09.859552 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.859480 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" Jul 24 13:42:09.859552 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.859518 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" event={"ID":"1d493ddf-88f1-4f51-9be5-da2d64442fcc","Type":"ContainerDied","Data":"991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc"} Jul 24 13:42:09.859683 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.859573 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z" event={"ID":"1d493ddf-88f1-4f51-9be5-da2d64442fcc","Type":"ContainerDied","Data":"aff2deb14e646c8539d92bf51bc4bdc9eaeebe5d5943ac0e7035c1607ee10ee9"} Jul 24 13:42:09.867185 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.867005 2554 scope.go:117] "RemoveContainer" containerID="655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a" Jul 24 13:42:09.867349 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:42:09.867320 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a\": container with ID starting with 655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a not found: ID does not exist" containerID="655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a" Jul 24 13:42:09.867410 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.867358 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a"} err="failed to get container status \"655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a\": rpc error: code = NotFound desc = could not find container \"655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a\": container with ID starting with 655b9c22accb7e4ff11dd898bafaafde917a1ff2081f1bc0c0c122b86de8741a not found: ID does not exist" Jul 24 13:42:09.867410 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.867382 2554 scope.go:117] "RemoveContainer" containerID="991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc" Jul 24 13:42:09.875106 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.874916 2554 scope.go:117] "RemoveContainer" containerID="991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc" Jul 24 13:42:09.875203 ip-10-0-134-100 kubenswrapper[2554]: E0724 13:42:09.875187 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc\": container with ID starting with 991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc not found: ID does not exist" containerID="991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc" Jul 24 13:42:09.875245 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.875208 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc"} err="failed to get container status \"991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc\": rpc error: code = NotFound desc = could not find container \"991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc\": container with ID starting with 991d8ef0fbcda2c2396d14ac346ab4476bce101fa2f9ba057a02519ee27812cc not found: ID does not exist" Jul 24 13:42:09.884403 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.884380 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z"] Jul 24 13:42:09.889642 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.889324 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b1a3df-7bd9965596-t624z"] Jul 24 13:42:09.904758 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.904736 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt"] Jul 24 13:42:09.909799 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:09.909775 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b1a3df-7bd69c7b86-qd7tt"] Jul 24 13:42:10.598437 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:10.598405 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d493ddf-88f1-4f51-9be5-da2d64442fcc" path="/var/lib/kubelet/pods/1d493ddf-88f1-4f51-9be5-da2d64442fcc/volumes" Jul 24 13:42:10.598783 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:10.598768 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d862a456-95ed-4f15-a6dc-73bee61f4f3f" path="/var/lib/kubelet/pods/d862a456-95ed-4f15-a6dc-73bee61f4f3f/volumes" Jul 24 13:42:11.660671 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:11.660638 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78d5j"] Jul 24 13:42:13.275265 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:13.275202 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:13.654331 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:13.654295 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:21.588007 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:21.587956 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:21.712147 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:21.712120 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:21.871766 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:21.871685 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:22.100232 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:22.100198 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:22.522760 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:22.522730 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b84db64b6-2nd7z_4cffa832-4fb9-496a-8a30-28be942b369e/authorino/0.log" Jul 24 13:42:28.892051 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:28.891973 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bd76cdcf-w4fk7_df987390-5a29-449a-b657-02ee9407f4ab/copy-manifests/0.log" Jul 24 13:42:28.906739 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:28.906715 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bd76cdcf-w4fk7_df987390-5a29-449a-b657-02ee9407f4ab/manager/0.log" Jul 24 13:42:30.952665 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:30.952607 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b84db64b6-2nd7z_4cffa832-4fb9-496a-8a30-28be942b369e/authorino/0.log" Jul 24 13:42:31.663653 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:31.663627 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-78d5j_42e606b6-b852-44ba-b52c-b8d77b32eb79/limitador/0.log" Jul 24 13:42:32.124079 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:32.124047 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:32.411856 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:32.411773 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6cb77c89-cfxxq_8d8a3e29-cb8c-4ddd-97c8-c782972a6722/kube-auth-proxy/0.log" Jul 24 13:42:33.495575 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:33.495544 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-svwvl_c61e409e-839e-44a5-a0fc-558f5b674c14/main/0.log" Jul 24 13:42:33.957164 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:33.957136 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-74kk6_d812304d-423c-4696-bbc4-0b9092529f9d/main/0.log" Jul 24 13:42:39.713601 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:39.713570 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:39.817983 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:39.817949 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:42.569087 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:42.569061 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ltm9n_cbea5b3b-f397-4df0-a942-9b238c3d15aa/global-pull-secret-syncer/0.log" Jul 24 13:42:42.674469 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:42.674433 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kpqcc_32792c2d-0873-44aa-9312-9b7d434bd6b8/konnectivity-agent/0.log" Jul 24 13:42:42.796393 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:42.796345 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-100.ec2.internal_7de245839c2e01ba31555968cc0536f8/haproxy/0.log" Jul 24 13:42:47.600757 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:47.600721 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b84db64b6-2nd7z_4cffa832-4fb9-496a-8a30-28be942b369e/authorino/0.log" Jul 24 13:42:47.971910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:47.971823 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-78d5j_42e606b6-b852-44ba-b52c-b8d77b32eb79/limitador/0.log" Jul 24 13:42:50.066520 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.066496 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-9dp5g_34abfc3c-d51e-4715-84fc-e7dc20268fb7/monitoring-plugin/0.log" Jul 24 13:42:50.311344 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.311271 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rz6fc_e8a9119e-fb00-42e5-8e4b-71039aeaf40a/node-exporter/0.log" Jul 24 13:42:50.338591 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.338550 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rz6fc_e8a9119e-fb00-42e5-8e4b-71039aeaf40a/kube-rbac-proxy/0.log" Jul 24 13:42:50.366286 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.366265 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rz6fc_e8a9119e-fb00-42e5-8e4b-71039aeaf40a/init-textfile/0.log" Jul 24 13:42:50.523403 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.523376 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af80fbc3-866f-4db1-bfa4-dfba42a6fc22/prometheus/0.log" Jul 24 13:42:50.551408 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.551379 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af80fbc3-866f-4db1-bfa4-dfba42a6fc22/config-reloader/0.log" Jul 24 13:42:50.579647 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.579583 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af80fbc3-866f-4db1-bfa4-dfba42a6fc22/thanos-sidecar/0.log" Jul 24 13:42:50.608910 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.608880 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af80fbc3-866f-4db1-bfa4-dfba42a6fc22/kube-rbac-proxy-web/0.log" Jul 24 13:42:50.635944 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.635925 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af80fbc3-866f-4db1-bfa4-dfba42a6fc22/kube-rbac-proxy/0.log" Jul 24 13:42:50.668543 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.668522 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af80fbc3-866f-4db1-bfa4-dfba42a6fc22/kube-rbac-proxy-thanos/0.log" Jul 24 13:42:50.694421 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:50.694401 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af80fbc3-866f-4db1-bfa4-dfba42a6fc22/init-config-reloader/0.log" Jul 24 13:42:52.084752 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.084719 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4"] Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.084994 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02063e5e-9e36-4141-9b29-a4c334b8f007" containerName="payload-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085005 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="02063e5e-9e36-4141-9b29-a4c334b8f007" containerName="payload-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085017 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d493ddf-88f1-4f51-9be5-da2d64442fcc" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085023 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d493ddf-88f1-4f51-9be5-da2d64442fcc" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085029 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="830a8ef3-595f-41a8-8e9f-338b8fd10bff" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085036 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="830a8ef3-595f-41a8-8e9f-338b8fd10bff" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085046 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e601e9-e515-4401-ac4f-20341c90e14e" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085051 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e601e9-e515-4401-ac4f-20341c90e14e" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085058 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f053ba3-c11d-446c-a33f-70f714678f42" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085063 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f053ba3-c11d-446c-a33f-70f714678f42" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085072 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fed51bd-253c-4b87-8fcf-8321299b2017" containerName="payload-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085076 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fed51bd-253c-4b87-8fcf-8321299b2017" containerName="payload-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085082 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5c42a4-d2a9-4a21-b8db-a511688c6418" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085087 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5c42a4-d2a9-4a21-b8db-a511688c6418" containerName="payload-pre-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085095 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3300c4-d9a6-4325-a8a1-6ae20d07d773" containerName="payload-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085100 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3300c4-d9a6-4325-a8a1-6ae20d07d773" containerName="payload-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085105 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d862a456-95ed-4f15-a6dc-73bee61f4f3f" containerName="payload-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085110 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d862a456-95ed-4f15-a6dc-73bee61f4f3f" containerName="payload-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085115 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d43c41-f470-4a67-9a45-6467bfdffb7d" containerName="payload-processing" Jul 24 13:42:52.085152 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085120 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d43c41-f470-4a67-9a45-6467bfdffb7d" containerName="payload-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085163 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d493ddf-88f1-4f51-9be5-da2d64442fcc" containerName="payload-pre-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085171 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e5c42a4-d2a9-4a21-b8db-a511688c6418" containerName="payload-pre-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085176 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="12d43c41-f470-4a67-9a45-6467bfdffb7d" containerName="payload-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085181 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="02063e5e-9e36-4141-9b29-a4c334b8f007" containerName="payload-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085187 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f053ba3-c11d-446c-a33f-70f714678f42" containerName="payload-pre-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085191 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="830a8ef3-595f-41a8-8e9f-338b8fd10bff" containerName="payload-pre-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085196 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e601e9-e515-4401-ac4f-20341c90e14e" containerName="payload-pre-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085201 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c3300c4-d9a6-4325-a8a1-6ae20d07d773" containerName="payload-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085206 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d862a456-95ed-4f15-a6dc-73bee61f4f3f" containerName="payload-processing" Jul 24 13:42:52.085850 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.085211 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fed51bd-253c-4b87-8fcf-8321299b2017" containerName="payload-processing" Jul 24 13:42:52.088082 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.088067 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.091297 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.091275 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qx26j\"/\"openshift-service-ca.crt\"" Jul 24 13:42:52.091430 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.091320 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qx26j\"/\"kube-root-ca.crt\"" Jul 24 13:42:52.092466 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.092440 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qx26j\"/\"default-dockercfg-mg26v\"" Jul 24 13:42:52.097162 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.097139 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4"] Jul 24 13:42:52.219345 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.219303 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6ht\" (UniqueName: \"kubernetes.io/projected/1887481b-e95e-4c30-ae20-3c0c1a5d333c-kube-api-access-ms6ht\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.219647 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.219358 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-sys\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.219647 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.219380 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-lib-modules\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.219647 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.219464 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-proc\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.219647 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.219530 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-podres\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.320810 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.320773 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6ht\" (UniqueName: \"kubernetes.io/projected/1887481b-e95e-4c30-ae20-3c0c1a5d333c-kube-api-access-ms6ht\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.320976 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.320828 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-sys\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.320976 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.320853 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-lib-modules\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.320976 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.320875 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-proc\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.320976 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.320913 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-podres\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.321148 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.320999 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-proc\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.321148 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.321021 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-lib-modules\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.321148 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.321029 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-podres\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.321148 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.321002 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1887481b-e95e-4c30-ae20-3c0c1a5d333c-sys\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.331658 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.331638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6ht\" (UniqueName: \"kubernetes.io/projected/1887481b-e95e-4c30-ae20-3c0c1a5d333c-kube-api-access-ms6ht\") pod \"perf-node-gather-daemonset-bk6v4\" (UID: \"1887481b-e95e-4c30-ae20-3c0c1a5d333c\") " pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.399259 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.399184 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:52.536798 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:52.535823 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4"] Jul 24 13:42:52.539380 ip-10-0-134-100 kubenswrapper[2554]: W0724 13:42:52.539350 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1887481b_e95e_4c30_ae20_3c0c1a5d333c.slice/crio-5c110129ecae5416fb682892b703fc3fa5ec5155ec3c8eca618d827eb72d8078 WatchSource:0}: Error finding container 5c110129ecae5416fb682892b703fc3fa5ec5155ec3c8eca618d827eb72d8078: Status 404 returned error can't find the container with id 5c110129ecae5416fb682892b703fc3fa5ec5155ec3c8eca618d827eb72d8078 Jul 24 13:42:53.042094 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:53.042058 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" event={"ID":"1887481b-e95e-4c30-ae20-3c0c1a5d333c","Type":"ContainerStarted","Data":"7c863cb47aa24e129ece55b3b571c2c02ffb45d5c07a57afd4426a63904efe43"} Jul 24 13:42:53.042287 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:53.042120 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" event={"ID":"1887481b-e95e-4c30-ae20-3c0c1a5d333c","Type":"ContainerStarted","Data":"5c110129ecae5416fb682892b703fc3fa5ec5155ec3c8eca618d827eb72d8078"} Jul 24 13:42:53.042287 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:53.042192 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:53.061458 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:53.061412 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" podStartSLOduration=1.061398683 podStartE2EDuration="1.061398683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:53.061050511 +0000 UTC m=+2989.064352634" watchObservedRunningTime="2026-07-24 13:42:53.061398683 +0000 UTC m=+2989.064700768" Jul 24 13:42:55.350692 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:55.350664 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mw26d_e8e1b35c-102a-43c8-b556-b4ba82ac1796/dns/0.log" Jul 24 13:42:55.375543 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:55.375514 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mw26d_e8e1b35c-102a-43c8-b556-b4ba82ac1796/kube-rbac-proxy/0.log" Jul 24 13:42:55.452078 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:55.452053 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-99gm8_3777315e-9137-4504-b546-c35d39f4613b/dns-node-resolver/0.log" Jul 24 13:42:56.114526 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:56.114497 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sx82n_d5073278-ac92-4fcd-a3bf-236f10e39dad/node-ca/0.log" Jul 24 13:42:57.459423 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:57.459400 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6cb77c89-cfxxq_8d8a3e29-cb8c-4ddd-97c8-c782972a6722/kube-auth-proxy/0.log" Jul 24 13:42:58.482671 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:58.482639 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8pq82_fb1b5900-c451-4a39-b80e-64ee6581fb10/serve-healthcheck-canary/0.log" Jul 24 13:42:59.055536 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:59.055510 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qx26j/perf-node-gather-daemonset-bk6v4" Jul 24 13:42:59.269446 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:59.269418 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgx2s_e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8/kube-rbac-proxy/0.log" Jul 24 13:42:59.300893 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:59.300865 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgx2s_e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8/exporter/0.log" Jul 24 13:42:59.332147 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:42:59.332068 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hgx2s_e5638423-8fd6-4c2a-bc1b-a3ad4c8943e8/extractor/0.log" Jul 24 13:43:02.484055 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:43:02.483973 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6cb77c89-cfxxq_8d8a3e29-cb8c-4ddd-97c8-c782972a6722/kube-auth-proxy/0.log" Jul 24 13:43:02.657676 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:43:02.657637 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bd76cdcf-w4fk7_df987390-5a29-449a-b657-02ee9407f4ab/manager/0.log" Jul 24 13:43:02.681522 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:43:02.681496 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bd76cdcf-w4fk7_df987390-5a29-449a-b657-02ee9407f4ab/copy-manifests/0.log" Jul 24 13:43:04.088053 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:43:04.088020 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-756cc79bf9-vl2jk_4e7ef589-9e7f-43bd-8296-c75224f5baeb/manager/0.log" Jul 24 13:43:04.378256 ip-10-0-134-100 kubenswrapper[2554]: I0724 13:43:04.378177 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6cb77c89-cfxxq_8d8a3e29-cb8c-4ddd-97c8-c782972a6722/kube-auth-proxy/0.log"